How to Build Evals for an AI Agent From Its Traces

Reviewing traces to build evals for an AI agent

A $320 hotel night in Port Selene should pass an expense check, since policy there allows $350. An expense-review agent sent it to a human anyway. The tools were right, the rule it retrieved was right and the reply read fine, so each standard check passed.

To build evals for an AI agent that catch this kind of miss, begin with what the agent actually did. A seven-step method works from its traces and ends by testing the evals themselves.

The test agent

The example is a LangGraph expense reviewer with five tools and a ten-section policy. It takes one expense and recommends approve, deny or human review. The test set had 42 requests, each run three times for 126 runs.

A wooden sample tray with compartments of salt, lentils, peppercorns, lavender, rice and green glass beads
Image: HITL Systems

Step 1: List who depends on it

Correct answers are only one duty. Write down everyone who relies on the agent and the log evidence each would need. Here the agent must save its decision once per review. In 23 of 126 runs it didn’t, or saved it twice, so the employee got an answer while finance got nothing to act on.

Wrong decisions occurred in 9 runs, 7.1%. The missing-record problem hit 18.3%, about 2.6 times more often. A grader checking only reply and policy would have scored the agent around 93% and missed it.

A brass jewelerโ€™s loupe and tweezers beside two cut emerald gemstones on a felt pad
Image: HITL Systems

Step 2: Design cases from dimensions

Asking an AI for 150 queries in one go produces duplicates. Instead, decide what to vary (who asks, which rule applies, what tools return), have an assistant propose values, review them, then generate one request per chosen combination. “Employee, hotel, missing receipt” becomes a request about a lost receipt.

A turntable needle resting in the groove of a plain black record on a green felt mat
Image: HITL Systems

Step 3: Capture full traces

A useful trace holds the request and reply, each tool call with arguments, results and errors, each model call with tokens and latency, and the outcome of any action the agent claims. Any tracing tool works. Open one trace before the full run and confirm you can follow it end to end.

Read all of a designed set of 100 to 150 cases. For production traffic of a thousand-plus requests a day, sample 150 to 200 traces at random from the past week, and pull targeted cases only when chasing a rare bug.

Step 4: Name the failure modes

“Wrong answer” doesn’t say what broke. Write a short note per mistake, with the evidence, then group the notes. The hotel note: the amount check returned the $250 standard cap, but Port Selene allows $350. A second trace sent a $240 dinner for two to review against a $150 per-person limit, though each share was $120. Both are one mode, a limit that doesn’t fit the case.

The 9 wrong decisions covered 7 requests. Repeats of the same request share one cause, so treat the real count as nearer 7, and treat a mode with a count of 2 as a lead, not a rate.

Step 5: Pick code or a judge

Ask what in the trace would show the failure. If it’s a field, a status or a pattern, write code: it’s free, deterministic and runs on every trace. If it takes reading, such as whether an explanation misleads, use an LLM judge, which costs a call per trace and needs checking against human labels.

The record check compares the decision in the reply with what the save tool stored. It passed 103 of 126 runs. Of the 23 failures, 21 never saved anything and 2 saved a different decision, so counting calls mattered more than reading wording.

Step 6: Test the evaluator

An evaluator that misses the failure will report that a bad fix worked. One check here originally matched wording in the reply. It was rewritten as arithmetic against the policy caps, since wording varied run to run and arithmetic didn’t. Give each evaluator four tests: a trace that should fail, one that should pass, and a paraphrase and a negation of each.

For a judge, write a pass or fail rubric for one mode, have a policy expert label traces blind, tune on the disagreements, then report two numbers on fresh data: the share of real failures it caught and the share of acceptable replies it passed.

Step 7: Change one thing and replay

The hotel bug sat in the amount tool, which applied one cap with no unit or city rate. The fix was to return the cap, its unit and the per-unit amount, leaving prompt and model alone. List the cases that should flip first. The $250 case matters because it sits exactly on the standard cap. With no standard-city night above $250 in the set, a lazy fix that lifts every cap to $350 would pass everything, so add that case.

Replay against a recording: the agent’s code reruns, and tools answer from saved results, except the one you changed, which runs live. After the rerun, sort every case as fail to pass, pass to pass, pass to fail or fail to fail, and read regressions first.

The result was mixed. Five cases went from wrong to right and none reversed. But four runs newly answered the employee without saving a decision. Port Selene already passed in the baseline, and one run per case can’t separate a fix from variation, so run each case three times.

Start this week

Pull 30 traces, write one note each and count failure modes before choosing metrics. Then write one evaluator per mode, add the four tests, and decide which cases must flip before you touch the agent.

Source: HITL Systems

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.