Coding Agent Evals: Build Them From Your Own Code

Coding agent evals built from a team's own git history

Once coding agents write most of the code, teams need to know whether each new model, rule or skill actually helps on their own codebase. The answer, according to one detailed guide, is to build coding agent evals from your own repository history and to treat the eval itself as something that needs testing.

The software factory problem

A growing number of teams run what amounts to a software factory, where agents take tickets, produce changes, run tests and open pull requests, frequently in parallel. On the Claude Code team, about 90% of the code is now written and reviewed by Claude.

Verification gates each change before merge, but the factory needs its own check. A model swap arrives every few months, a CLAUDE.md rule on Tuesday, a skill on Thursday, an overnight agent update. Each one shifts every pull request that follows, and hardly anyone measures the effect.

Public leaderboards can narrow a model shortlist, yet they run on other people’s code. A factory needs evals taken from its own work.

Start with your git history

Every team already has real work with known answers:

  • Merged pull requests: a genuine request, the prior code and an approved change.
  • Reverts and hotfixes, the richest source, because each one marks a failure that slipped through review.
  • Bug reports and failed agent sessions, which record what developers wanted and where the agent stumbled.

To turn one into an eval, check out the commit before the fix, give the agent the developer’s original wording and score the output with the tests from the real fix. Recent work has another advantage: a fix from last month is far less likely to sit in a model’s training data than a famous open-source fix from three years ago.

Why a fair eval is hard

Converting a pull request into an eval seems quick, but early attempts often measure the wrong thing. Typical problems:

  • The solution leaks via git history, a newer dependency or a web search.
  • The environment won’t build, or tests were already red at that commit, so the agent is blamed unfairly.
  • The prompt hands over the fix: “fix the null check in parser.py” is a far easier task than “imports crash on empty files.”
  • The tests pin one specific implementation, so another valid solution fails.
  • The agent cheats the grader by rewriting or skipping tests, or hard-coding the failing input.
  • The grader has the same blind spots as the agent when similar models write both code and checks.

The remedy is to validate the eval first. Make sure the answer can’t be reached. Then replay known solutions in the real environment: the actual fix passes, an alternative correct fix passes, and an untouched repo fails. Feed it bad attempts too, such as a wrong patch, an empty one and a half-hearted one. A reviewer who didn’t build the eval should look it over, and the eval should be frozen before its first real run so edits show up. No model is needed for any of this.

Measure the factory, not only the output

Trusted evals turn each setup change into an experiment: run identical tasks before and after swapping a model, rule, skill or hook. Confirm the change took effect, because a rule in AGENTS.md never reaches an agent that reads only CLAUDE.md.

Repeat every task several times, since agents seldom give the same result twice. Report solve rates per setup and admit when the gap is within noise. Record the process too: did it run tests, stay in scope, describe its work honestly? Show cost and time per solved task beside the pass rate, because identical pass rates can hide big price differences.

Split results by type of work. An agent might handle test fixes well and fumble migrations, and that split decides what runs unattended, what gets reviewed and what stays human.

Keep the suite fast and current

Eval suites bloat the way test suites do: incidents add tasks and none are removed. Each task builds a container and runs an agent for minutes, so a full pass takes hours, and people stop running a suite that slow.

  • Tag each eval with the area it covers, run the matching ones plus a small fixed sample of the rest, and keep the full suite for rollouts.
  • Drop tasks that every setup passes or none does, and note the reason.
  • Build environments once and reuse them, as setup frequently takes most of a trial’s time.
  • Re-check evals when requirements change and keep adding tasks from recent work.

What is still unsolved

The author concedes the problem is far from solved: fair evals are costly, comparisons are noisy and suites go stale. The questions put to teams running agents at scale are how they judge that a change helped, how they keep evals up to date and when their numbers misled them. The answers will set how much of the factory can safely run unsupervised.

Source: X (formerly Twitter)

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.