The Epoch Capabilities Index, or ECI, has become one of the most watched charts in AI. Developed by the nonprofit Epoch AI, it condenses many benchmark results into a single capability number per model. The latest scores sit around 169.
Epoch researcher Alexander Barry and Florian Brand, who works on open models at Prime Intellect and has contracted for Epoch, recently walked through how the Epoch Capabilities Index is built and where it can mislead.
How the index is built
Barry says the index uses a relatively simple statistical model. It takes many benchmark results and outputs one score for each model. It rests on item response theory, which assumes a single quantity drives performance across all of a model’s benchmark scores. That is the same logic behind human IQ tests.
The design is continuously updatable. When benchmarks saturate or drop out and new ones appear, the new data can be added and results stay on one scale. That makes it possible to look at trends going back years and forecast future models.
The scale is arbitrary by definition. The original Claude 3.5 Sonnet is set at 130 and GPT-5 at 150, so a 20-point gap equals the distance between those two. Barry expects the top of the scale to reach 170 soon, which would be the same distance again above GPT-5. Models typically gain about 15 points a year.

Is intelligence really one dimension?
Barry concedes the single-dimension assumption isn’t literally true. Models have relative strengths, and one may be better at math than at something else. Still, he calls it very close to true empirically. A new flagship is usually better than a six-month-old model on nearly every benchmark. Domain versions covering only math or only software engineering tend to differ from the general score by about three to five points at most.
So models can be jagged compared with humans without being jagged compared with each other.
Barry prefers this approach to METR’s time-horizon method, which measures the length of tasks a model can finish at a given accuracy. Time horizons need costly human baseline times for every task, and for long tasks that may become impractical. The ECI needs only benchmark results.

The benchmark selection problem
Brand argues the math is hard to fault but the inputs are not. A single index is easy to understand, he says, yet the unbounded number is hard to put in context. A benchmark capped at 100 shows how much room is left. An ECI score does not.
Weighting follows from the model, in his view, but choosing which benchmarks go in is subjective, and including or excluding scores can push results in different directions. Barry says the index includes capability measures, not things like alignment. It also needs benchmarks whose ceiling is known. A test that seems to allow 100% but tops out around 65 can distort estimates. Epoch has removed benchmarks it later found were biasing results.
Brand’s example is SimpleQA Verified, a factual-recall test. Open models have historically done worse on it, and bigger models that store more obscure facts score higher. Barry notes that model size shows up there, and says the benchmark’s influence has shrunk. Smaller recent Anthropic models, for instance, are surprisingly strong in many areas but lag on knowledge tests.

Epoch’s own benchmarks and open models
Barry acknowledges that many included results come from Epoch-run benchmarks, with a historic lean toward math. He adds that labs tend to highlight benchmarks that flatter their models, which can make a release look better than a broader set suggests.
To address this, Epoch offers math and software-engineering versions, plus custom indexes. Anyone can ask for the score without SimpleQA.
That matters for open models. Brand says the software-engineering index matches his experience better, and the open-versus-closed gap is smaller there. Barry says Kimi K3 scores two to three points higher on that index than on the general one, and is roughly level with Opus 4.8 either way.
What to take from the number
A gap of two or three points, Barry suggests, is less a verdict than a rough signal. Debates over 165 versus 172 treat an arbitrary scale as exact. The Epoch Capabilities Index works best as a trend line, with the sub-indexes consulted when a particular skill matters.
The open question, raised in the discussion, is whether static benchmarks stay useful as models improve.
Source: asteriskmag.substack.com