Staff Software Engineer - ML Infrastructure

About the Role

At Watney, ML Infrastructure engineers turn data collected from a live fleet of robots into better models. The fleet produces large volumes of video and telemetry data from real work in the field, and making that data trainable is one of the hardest systems problems at the company.

As we continue to scale, these systems will require larger training runs with more data, expanded clusters, and optimal GPU utilization.

What You’ll Do

• Own training and inference infrastructure.
• Build the data pipelines that these training runs depend on.
• Make experiments fast to launch and reproduce.
• Contribute to our core training code

.

You May Be a Good Fit If You:

• Have built ML infrastructure that carried real production training runs.
• Have scaled distributed training systems.
• Strong experience with Python, PyTorch or TensorFlow.
• Have experience identifying and troubleshooting GPU performance bottlenecks in large-scale training environments

.