True Talent Solutions Partners

Research Engineer, Evals

San Francisco
Salary
$250,000 - $290,000 annually
Level
Mid
Work type
On-site · San Francisco
Visa
Sponsors visas

Open. First seen 3 November 2026.

Apply on the company's site →

About the role

About the Role

Build evaluation systems that measure the quality of web data extraction across diverse sites, formats, and edge cases. You will design the metrics, datasets, and pipelines that turn quality findings into model and product decisions.

What You'll Do

  • Define rigorous metrics for evaluating structured extraction quality across varied web content.
  • Build scalable evaluation pipelines and harnesses to measure system performance.
  • Generate and curate ground-truth datasets for trustworthy evaluation.
  • Connect evaluation results to model and product improvements, closing the feedback loop.

What We're Looking For

  • Typically 4 to 10+ years of relevant experience, including at least 2 years working on evaluations.
  • Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.
  • Experience building evaluation systems for LLM extraction of structured fields from messy, real-world web pages or documents, at large scale.
  • Ability to design evaluation methodology and systems from the ground up, rather than only running existing benchmarks.
  • Statistical rigor in metric design and interpretation, including sampling, estimator bias and variance, error bounds, significance, and judge-to-human agreement.
  • Experience creating gold-label datasets and using evaluation results to guide model or product decisions.
  • A blend of research depth and engineering ability, with comfort tackling ambiguous problems and iterating quickly.

Compensation & Benefits

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, based in San Francisco, California. Toronto, Canada is also an available work location. The role requires working on-site five days per week in either city.

What this role asks for

Particular to this role

NLP

Rare in True Talent Solutions Partners's other 123 open postings.

Standard for software engineering roles

Python

Common in software engineering postings at any company.

Compared with True Talent Solutions Partners's other open jobs.

The team: Client Roles

This role is one of 127 open in Client Roles. 127 were posted in the last 30 days, including 7 lead roles: the team is hiring the people who run it too.

WHAT THE TEAM WORKS WITH, ACROSS ITS POSTINGS

Python (66 of 127), LLM (48 of 127), TypeScript (46 of 127), React (44 of 127), JavaScript (27 of 127), Machine Learning (27 of 127), AWS (25 of 127)

OTHER ROLES ON THIS TEAM

From the team and dates on True Talent Solutions Partners's own job board.

Posting history

Reposted 2 times since 30 October 2026: a posting for this role in San Francisco was taken down and the role was posted again.

Similar jobs

More jobs like this

SKILLSPython 46,831Machine Learning 19,854LLM 18,638NLP 2,443
PLACESSan Francisco 7,730United States 73,452
COMPANYJobs at True Talent Solutions Partners 127

Is this your posting? Email info@careerholo.com to have it removed.