METR Time Horizon Update: AI Struggles With Judgment

Chart showing the METR time horizon update for frontier AI models

The METR time horizon plot, one of the most widely shared and most misread graphs in AI, has been updated, and it shows a split. The evaluation nonprofit METR finds that some AI capabilities still follow an exponential trajectory. Models are weaker, though, on tasks that depend on judgment than on coding.

How the original plot worked

METR had models attempt a large suite of tasks that people had also completed. Each task’s difficulty was measured by how long the human testers took, and each model was then rated by the length of tasks it could finish. The result was striking: every four months, the length of tasks frontier models could complete successfully doubled.

The plot spread widely, and many readers misinterpreted it because the measure is complicated.

What the update adds

The new version includes more tasks. Some are estimated to take humans far longer than anything in the first suite, and others are more open-ended. That matters because of a central question in AI: whether progress on verifiable tasks carries over to tasks where success is hard to measure.

Verifiable tasks are those where software can check automatically whether the AI succeeded. Researchers train models on them with reinforcement learning, repeatedly attempting a task, strengthening what led to success and weakening what led to failure. That is why AI is so good at coding and math. The technique is much harder to apply when success is fuzzy or slow to confirm, as when choosing a new research direction. The original plot left such tasks out.

Where the models fall short

METR added some judgment-heavy tasks that bear on automating AI research and development. In one, models design new AI evaluations that METR could use, and METR staff score them. Though the top models now beat almost every human at software engineering, not one AI-designed evaluation was good enough for METR employees to find it useful.

In another task, models score philosophical arguments, which is meant to capture good judgment. METR says its employees disagree about how relevant that task is to automating AI research. Models fell short of human performance there as well.

There are signs of movement. Some of the newest models, including Opus 5.5, tend to reach what METR calls “borderline above the bar” on the evaluation design tasks, meaning not yet usable but promising. A year ago, every model landed in the “clearly below the bar” category.

Why researchers care

The question is live because a heavily redacted METR report earlier this month claimed AI might be doubling the pace of AI progress at Anthropic. That prompted fears the company, which recently had a large IPO, could soon reach full recursive self-improvement, where AI designs its own successors without human involvement.

Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, welcomed the change. “We’re at the point where AI evaluations need to get more creative,” he said. He wasn’t surprised by the weak results, since an August preprint from him and collaborators found that models can’t yet reproduce human computer science research on their own. OpenAI said in a Wednesday blog post that its models do worse on real tasks assigned by employees than on the easier, verifiable ones in METR’s original suite.

Opinions differ on what the gap means. Kapoor thinks research taste, the ability to pick promising questions and experiments, will be a major bottleneck. Others argue that models trained on easy-to-check tasks will eventually handle open-ended ones too.

Limits of the new measurements

The new tasks are hard to design and score, so the report includes few of them. Most completion times are estimates, because paying people to do tasks that take hundreds of hours, or that few experts can do, is too costly and slow. For the philosophy task, one person rated most of the arguments, so models are judged on agreement with that individual.

METR staff member Matt Putz said trends for these tasks are unlikely to become as clean as those for easy-to-specify ones. Najoung Kim of Boston University said the gains tell us something about now, but she isn’t sure how they extrapolate.

What comes next

Experts see the shift toward open-ended tasks as progress, yet agree that definitive answers on automated AI research are far off. Better judgment-based evaluations, with more raters and measured human times, are what would tighten the picture.

Source: MIT Technology Review

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.