AI forecasters now equal or beat top human performance on traditional forecasting questions, according to benchmarks such as ForecastBench and the Metaculus Cup. The strongest are specialized systems from startups like FutureSearch, Mantic and Preseen. General models such as ChatGPT and Claude trail them by about six months, so they may match top humans by the end of the year.
Why this changes the options
Before this, high-quality forecasts were slow or limited. A team of superforecasters costs thousands of dollars and takes weeks. Prediction markets and Metaculus cover only a small set of general-interest questions, and market resolution criteria can be gamed.
With traditional questions largely solved, the open task is learning how that success fades on other kinds of questions. A conversation with experts at the Forecasting Research Institute (FRI) outlines six.
1. Conditional forecasts
A traditional question asks whether a party wins an election. A conditional one asks what happens to a stock market crash if it does. Conditional questions speak directly to policy, but prediction markets have been slow to offer them, partly because stakes must be refunded if the condition fails. AI forecasters can take such questions now, and FutureSearch, Preseen and Mantic list it as a feature.
What’s unproven is quality. About a year ago FRI tested this and found the AIs of the time did significantly worse on conditional questions. The experts expect gradual improvement rather than a sudden switch. They think current systems are still worse than top superforecasters, and a benchmark due in a few months should show where they stand.
2. Causal forecasts
A conditional answer isn’t a causal one. If the economy is bad in 2027 whenever a party wins in 2026, the cause may be a bad economy that already existed. The experts pointed out that a causal forecast is a combination of conditional forecasts across several variables, such as prior economic conditions or a war. Humans can’t forecast hundreds of branches, but an AI could, at the cost of more API spend.
FRI plans to test this with Freeciv, a clone of the strategy game Civilization. A saved game can be replayed with one thing changed, so causal outcomes can be measured directly.
3. Forecasting AI progress
AI is an anomaly for forecasters. Top humans and AI superforecasters have underestimated its pace every year. Earlier this summer they gave only a 5% chance that AI would solve a Millennium Problem this year. Some experts see one correlated miss across 2022 to 2026 that should revert to the mean. Others think base-rate methods lose to first-principles ideas like scaling laws in this field.
Telling a model to adopt a fast-timelines view doesn’t work, the experts said. It just raises every AI-related estimate without working through the implications.
4. Long-term forecasts
Researchers and market bettors don’t want to wait fifty years for results, so evidence is mostly short term. The one study of twenty-five year forecasts covers humans, not AI. A clean test would strip modern knowledge from a model and ask it to forecast a century ahead. Talkie, a model trained on documents from 1931 and earlier, exists, but there isn’t enough old text for it to be capable.
5. Questions that never resolve
Some questions look like forecasts but concern the past, such as whether COVID came from a lab leak. Markets can ask what scientists will conclude in 2040, though that assumes scientists converge on the truth. The writer pressed an AI forecaster for its breakdown. It said 26% for a lab leak, 37% that the 2040 community would be convinced if true and 3% that it would be falsely convinced, for a 12% total. The writer couldn’t tell whether those figures reflect real skill or an answer made to satisfy the request.
6. Choosing the right questions
Skill at answering is only as useful as the questions asked. One idea is to have an AI generate a thousand cruxes for a big question, such as America’s status in 2100, and turn each into a conditional forecast. A crux whose two outcomes change the final probability most is the one worth watching.
What comes next
FRI’s conditional-forecasting benchmark is expected within months, and its Freeciv work will test causal methods. Those results will show whether the lead on traditional questions carries over.
Source: astralcodexten.com