Eval costs have become a reason some AI teams skip testing. Engineers report running fewer evals on development builds and scoring less production traffic as managers watch budgets. Benchmark results suggest the money goes to defaults. Teams pick the largest judge, feed it entire traces, leave reasoning on and score everything.
Each default has a cheaper alternative, and several held their accuracy in testing. Eight changes stand out.
Use code where code fits
A simple test decides it: if two reviewers using one rubric would always agree, write code instead of calling a model. Structure and format checks and matches against a known answer fit. Judging meaning, such as spotting a refusal, still needs a model.
On 1,500 MATH-500 answers, the Math-Verify library was 99.8% accurate at no cost, while a Sonnet 5.5 judge was 99.9% accurate at $3.46 per 1,000. Putting a schema check ahead of a judge on 890 tool calls settled 38% for free and lowered judge spend from $2.17 to $1.34 per 1,000 calls while accuracy held at 96.5%. Gains shrink where judgement dominates. On PII documents, pattern rules resolved just 18%.

Sample production traffic
You need enough traces to know a pass rate, not all of them. With 35% of responses hallucinating in the RAGTruth set, scoring 10,000 daily traces with Claude Opus 5 runs about $145 a day. A random 500 costs $7.27 and still pins the rate to within ยฑ4 points. Give those 500 to Haiku 5.5 and the bill is 9 cents.
Version-to-version comparisons need larger samples. Spotting a move from 35% to 40% takes about 1,500 traces per version, a 10-point jump about 380, and a 2-point change about 9,000. Guardrail evals hunting rare failures shouldn’t be sampled.

Match the eval level to the question
A tool-choice question needs one span, a looping question needs the trace, and a solved-problem question needs the session. In 199 web-agent traces, a single span carried roughly 7,700 tokens, against about 120,000 for all 18. Answering from one span cost roughly 1/16th as much.

Trim the input
Full-trace prompts are mostly filler, including system prompts, tool schemas, repeated history and page dumps. Keeping only the goal, the agent’s reasoning, its actions, any errors and the final page shrank an average trace from about 155,000 tokens to 6,500. Spend dropped 20x to 24x with no accuracy loss for any judge tested. Cut the boilerplate, but leave the evidence.
Try a decision model
A decision model returns probabilities for fixed options rather than writing out a verdict, so there are no output tokens to pay for. Tested on 1,338 labelled hallucination responses, one such model reached 85.4% for $0.046 per 1,000 evals, with median latency of 82 ms. Opus 5 at high effort reached 84.9% for $14.53 and needed 3.2 seconds. The approach suits classification and struggles with maths, code and chained reasoning.
Pick a cheaper judge and change how you call it
On hallucination, DeepSeek V4.1 Flash landed within 2 points of Opus at roughly 1/600th of the price. Haiku 5.5 over-flagged responses until its threshold was tuned against human labels, after which it hit 86.3%, matching Opus for about 1% of the price. Results don’t transfer between evals: GPT-6 Luna scored 99.9% on maths but sat 10 points behind Opus on hallucination.
Reasoning tokens bill as output, so start with reasoning off and enable it only where a test shows a gain. Turning it on dropped DeepSeek’s hallucination accuracy from 83.1% to 75.5% while multiplying cost by 7. Running Opus 5 at low effort with batch pricing and caching cut its price from $14.53 to $3.09 per 1,000, and accuracy edged up to 86.0%.
Add a cascade
A cascade lets a cheap judge handle the clear cases and passes only uncertain ones upward. With a decision model up front and 8% of cases going to Opus 5, accuracy reached 86.8% for $1.40 per 1,000, beating Opus alone at about a tenth of the cost. Sending the uncertain cases to Sonnet 5.5 instead gave 86.6% for $0.21.
Check the numbers on your own traffic
Results varied too much to copy blindly. Reasoning helped on PII detection and hurt on hallucination, and trimming helped one eval while hurting another. Pull a few hundred labelled examples, run the current judge beside two or three candidates, and compare accuracy with price. In these experiments such a check ran from a few cents to a few dollars per eval.
Two setups came out ahead. Development and CI did best with an inexpensive judge, reasoning off, batch billing and a compact dataset modeled on production. Production did best with code checks first, then an inexpensive judge, a cascade for doubtful cases, trimmed inputs at the narrowest level and a traffic sample.
Source: Arize AI