Teams building AI features keep asking whether they can replace A/B tests. A new analysis says it can work, but only under strong assumptions that are hard to verify and can change over time. Noise is a second trap, because it can hide a strong link between offline and online results.
In the worked example, a naive estimate says the eval explains 24% of the variation in online results. The true figure is 60%.
Two ways to test a change
When a team changes an AI feature, for instance with a new prompt or a different underlying model, there are two main ways to check that it improved. Evals of this kind are quick tests on a fixed set of sample prompts, run before any real user sees the change. Other LLMs often act as judges and grade the answers.
A/B tests instead give the new version to a random group of real users and compare their behavior with users on the old version. The speed difference is the draw: offline evals take hours, while A/B tests often take days or weeks.

What a safe decision requires
The aim is to decide from the eval while capping the chance of releasing something that hurts the online metric. Ideally a team would know how the true online effect is distributed for any observed eval effect. That tells it how likely a given result is to help or hurt real users.
Learning that relationship raises two challenges. First, the team needs a pool of paired offline and online results that all come from the same underlying relationship. If an eval gain means a real gain for some changes but something different, or the opposite, for others, the method breaks.
Second, both numbers are noisy. Offline scores carry sampling error from drawing examples out of a bigger set and from LLM grading. Online effects have their own sampling error.

Why measurement error fools teams
A simplified working model assumes a large, homogeneous set of past experiments, normally distributed true effects, zero-centered measurement errors with known variances, and judge errors that are independent of the treatment arm. Under those assumptions the relationship is fully specified.
R², the squared correlation, shows roughly how much of the online result the eval explains. High means informative. Low leaves uncertainty however precisely the eval is measured.
The naive estimator is pulled down by noise in both measurements. That is how a true 60% shows up as 24%. Ignoring measurement error means a team can write off a good proxy. Classical meta-analytic methods can model the error and correct for it.

A gate for offline evals with three outcomes
If the assumptions hold, a decision gate can be built. Sufficiently positive eval results lead to shipping, sufficiently negative results lead to killing the change, and everything between goes to an A/B test.
The thresholds depend on how strong the offline-online relationship is. With perfect surrogacy, the usual significance levels apply directly to the eval. The weaker the relationship, the more evidence the eval must show before anyone acts on it.
Where the assumptions break
Pooling different classes of changes can create Simpson’s-paradox effects. A strong relationship can appear in the pooled data even if the eval predicts nothing within any single class, so a team may badly overestimate how well the eval stands in for tests.
Judge bias matters in a specific way. A biased judge is fine if the bias is the same in both arms, because it cancels when you compare treatment with control. If the bias depends on the arm, it can distort the effect and even flip its sign, making a worse version look better. That is plausible when comparing different LLMs that power an agent, since judges may favor their own generations. Research cited with the analysis suggests part of that apparent self-preference can reflect real differences in quality.
Keeping the gate honest after launch
Once a gate is deployed, A/B tests stop for changes in the ship and kill regions, so the team loses ground truth where decisions get made. The relationship could drift or vanish without anyone noticing.
One remedy is an audit pool: randomly send a small share of would-be ships and kills to an A/B test anyway. That produces paired data across all regions and lets the team detect drift and update the gate. The cost is fewer decisions made without an A/B test.
So can teams rely on offline evals alone? In principle yes, if they accept strong assumptions and believe they are reasonable. In practice the bar is high, and the safest route keeps a stream of A/B results flowing to check that the offline signal still means what it used to.
Source: Spotify Engineering