Applied Scientist / Research Engineer — Data Quality & Evaluation

Job Description:

Job Overview

We are looking for an Applied Scientist/Research Engineer to improve the quality of training data and evaluation systems used to develop advanced AI models.

In this role, you will investigate how data quality, dataset composition, and evaluation methods influence model performance. You will design experiments, develop data curation and scoring methods, analyze model failures, and build tools that help us identify the data most valuable for training and improving AI systems.

This role is ideal for someone with a strong background in applied ML research, data-centric AI, or model evaluation who enjoys turning research insights into practical improvements.

What You Will Own

• Develop methods for training data curation, quality scoring, dataset selection, and data mixing.
• Design evaluation frameworks and benchmarks to measure model capabilities and identify performance gaps.
• Analyze model failures to understand how training data affects model behavior and generalization.
• Apply techniques such as active learning and human-in-the-loop evaluation to improve data efficiency and annotation quality.
• Build automated tools and pipelines to improve data quality while reducing annotation costs and manual effort.
• Design and run experiments to measure how data improvements translate into better model performance.
• Collaborate with research, engineering, and data operations teams to turn findings into scalable solutions.

Required Qualifications

• Experience in applied ML research, data science, AI evaluation, or a related technical field.
• Strong Python skills and experience with relevant ML frameworks and data processing tools.
• Experience with one or more of the following: training data curation, dataset optimization, model evaluation, benchmark development, active learning, or post-training.
• Strong experimental design and analytical skills, with the ability to connect data characteristics to model performance.
• Experience developing technical solutions, research methods, or automated pipelines rather than only performing annotation or data labeling.
• Ability to communicate research findings clearly and translate them into actionable improvements.

Preferred Qualifications

• Experience at AI data or model development companies (ex. Scale AI, Mercor, Surge AI, or similar organizations).
• Experience improving model performance through better training data or dataset composition.
• Experience building evaluation frameworks or analyzing model failures.
• Experience with human-in-the-loop systems, annotation quality measurement, or data selection algorithms.
• Research publications or demonstrated applied research contributions are a plus.