Senior Software Engineer, Machine Learning Infrastructure & Automation

About this role:

Help fal's ML team move faster by building the automation, infrastructure, and developer tooling that makes developing, testing, and deploying generative AI models seamless.

You'll own and improve the CI/CD systems supporting our rapidly growing collection of ML models and inference pipelines. Your focus will be on eliminating manual work, accelerating development cycles, and building reliable systems that allow ML engineers to ship new models and optimizations with confidence.

This is a high-impact engineering role where you'll work closely with our Applied ML and ML Performance teams. You'll build everything from automated model validation and performance benchmarking to AI-powered development workflows that help engineers iterate faster.

The ideal candidate thinks beyond traditional CI/CD and sees automation as a force multiplier for the entire engineering organization.

What you’ll do:

• Own ML CI/CD infrastructure Design, build, and maintain automated testing, validation, and deployment pipelines for our ML models and inference services.
• Accelerate development cycles.Dramatically reduce CI execution times through intelligent parallelization, caching, test selection, and efficient use of compute resources.
• Build automated model validation.Develop systems that test model outputs, detect quality regressions, and validate changes across different models, GPU architectures, and configurations.
• Automate performance benchmarking. Build continuous performance testing that detects regressions in inference latency, throughput, GPU utilization, and cost.
• Build automated pricing and deployment checks. Ensure model pricing, billing configurations, API schemas, and deployments are validated automatically before reaching production.
• Extend our agentic engineering workflows. Develop AI-powered automation and agentic coding systems that automatically diagnose CI failures, identify regressions, propose fixes, and streamline engineering workflows.
• Improve deployment reliability. Build automated safeguards, deployment verification, rollback mechanisms, and monitoring to ensure new model releases are reliable.
• Eliminate engineering toil. Identify repetitive tasks across the ML team and build tools and systems that automate them, allowing engineers to focus on developing new models and improving performance.

Qualifications/Nice-to-haves:

• 5+ years of software engineering background with proficiency in Python and experience building production infrastructure and developer tooling.
• Experience designing and operating CI/CD systems using GitHub Actions or comparable technologies.
• Deep understanding of automated testing, build systems, dependency management, caching, and parallel execution.
• Experience working with containerized workloads, Docker, and cloud infrastructure.
• Ability to design reliable distributed systems and debug complex infrastructure failures.
• Strong understanding of observability, including logs, metrics, tracing, and automated alerting.
• A passion for developer productivity and a demonstrated ability to eliminate manual processes through automation.
• Comfortable working independently, identifying high-impact problems, and building end-to-end solutions.
• Experience with ML infrastructure, PyTorch, GPU workloads, or model-serving systems.
• Familiarity with NVIDIA GPU architectures and multi-GPU environments.
• Experience building automated inference benchmarks or ML quality evaluation frameworks.
• Experience with agentic coding tools such as Codex or Claude Code, or building custom AI engineering agents.
• Experience optimizing CI/CD pipelines at scale, including distributed test execution and ephemeral compute environments.
• Experience developing internal developer platforms or infrastructure-as-code tooling.

Tech Stack:

• Python, PyTorch, Docker, GitHub Actions
• NVIDIA GPUs and distributed GPU infrastructure
• Model serving and inference pipelines
• Observability and performance monitoring tools
• AI coding agents and automated engineering workflows
• Access to fal's massive GPU cluster for testing and development