Senior Software Engineer, AI Evaluation Infrastructure

About Turing

Turing’s mission is to accelerate superintelligence to drive real economic progress. Headquartered in San Francisco, Turing works with frontier AI labs to generate high-quality datasets, reinforcement learning environments, and frontier research benchmarks that improve model capabilities in software engineering, enterprise knowledge work, and advanced STEM reasoning. In software engineering, Turing is the largest and longest-running data provider in the category. Turing also works with Fortune 500 enterprises across financial services, life sciences, healthcare, retail, automotive, and CPG to build and deploy end-to-end agentic AI systems inside mission-critical workflows. By operating on both sides, Turing closes the loop between frontier research and enterprise deployment, turning real-world deployment signals into better data, evaluations, and more capable models. Learn more at www.turing.com.

Senior Software Engineer, AI Evaluation Infrastructure

We build the infrastructure that frontier AI labs use to evaluate coding agents and language models: benchmark datasets, agent evaluation runs at scale, and automated task quality checks. You would own features across the stack, from the data model and background jobs to the screens our users work in every day.

This is a role for an experienced engineer who works independently, makes sound tradeoffs, and turns a loosely defined request into something shipped, tested, and safe to leave alone.

What you will do

• Design and ship features end to end: data model, API, background workers, and the UI when the change needs it.

• Build and improve how agents and models are evaluated: running trials in sandboxed environments, scoring results, and turning thousands of runs into results people can trust.

• Improve how benchmark tasks are quality-checked: verifiers, rubric-based and LLM-as-a-judge grading, and deciding whether a task is clear, solvable, and fair to score.

• Integrate new models, agents, and benchmarks in a way that generalizes, without a special case for each one.

• Keep the system fast and reliable as usage grows: efficient queries, safe migrations, and concurrency that holds up under load.

• Review teammates’ pull requests against our engineering guidelines and help keep the codebase consistent.

What you bring

• Typically 5+ years building and operating production backend systems, with ownership of at least one service from design through on-call.

• Strong Python. Experience with Django (or a similar framework), PostgreSQL, and a task queue such as Celery. What matters most is having run a real system, not knowing a particular framework.

• Hands-on experience in LLM evaluation, beyond calling a model API. You understand how a benchmark task is built, how an agent trial is scored, and how results are aggregated and compared. Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar harness is a strong plus.

• Judgment about task quality. You can spot ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason.

• Good tradeoffs. You reach for the smallest design that solves the problem, extend what already exists before adding something new, and can explain what you chose not to build.

• Ownership. You take a fuzzy request, find the relevant code, ask ques

Responsibilities

• Design and ship features end to end: data model, API, background workers, and the UI when the change needs it.
• Build and improve how agents and models are evaluated: running trials in sandboxed environments, scoring results, and turning thousands of runs into results people can trust.
• Improve how benchmark tasks are quality-checked: verifiers, rubric-based and LLM-as-a-judge grading, and deciding whether a task is clear, solvable, and fair to score.
• Integrate new models, agents, and benchmarks in a way that generalizes, without a special case for each one.
• Keep the system fast and reliable as usage grows: efficient queries, safe migrations, and concurrency that holds up under load.
• Review teammates’ pull requests against our engineering guidelines and help keep the codebase consistent.

Requirements

• Typically 5+ years building and operating production backend systems, with ownership of at least one service from design through on-call.
• Strong Python. Experience with Django (or a similar framework), PostgreSQL, and a task queue such as Celery.
• Hands-on experience in LLM evaluation, beyond calling a model API, including understanding how a benchmark task is built, how an agent trial is scored, and how results are aggregated and compared.
• Judgment about task quality, including spotting ambiguous instructions, weak or gameable verifiers, leaked answers, and checks that pass for the wrong reason.
• Good tradeoffs, including reaching for the smallest design that solves the problem and extending what already exists.
• Ownership to take a fuzzy request, find the relevant code, and ask questions.

Preferred Qualifications

• Experience with Harbor, Terminal-Bench, SWE-bench, Inspect, or a similar harness.