Does an Intelligence Explosion Matter for AI Policy?

Debate over an intelligence explosion and pacing AI frontier progress

An intelligence explosion may be the wrong thing for AI policy to argue about. That is the case one economist and writer makes against the framing that dominates AI safety talk in San Francisco: whether recursive self-improvement is near, and whether labs should slow down to stop it.

The argument is that the AI we are already getting will look about as godlike, and matter about as much to the economy, as a system that suddenly bootstraps itself.

Where the debate stands

Sceptics make mostly empirical and economic points. Ramez Naam argued that OpenAI data show AI automating only short chunks of the research process. Diminishing returns mean each capability gain buys a smaller speedup, and faster progress needs far more compute and money. He sees no sign yet of acceleration. Theo Jaffee noted that AI hasn’t fully automated even small research tasks. Luis Garicano and Alex Imas found AI progress has sped up but at a higher cost per unit of progress, and their model suggests ideas are getting harder to find.

On the other side is a METR report on Anthropic’s internal progress in automating AI research. It relies on non-public data and an internal capability measure, so how much to trust it takes guesswork. Its basic story is that Claude is already speeding up Anthropic’s research a lot. The report’s authors guess that about 80% of the recent acceleration is due to AI, and not to faster growth in training compute.

Dwarkesh Patel argues that AI research resembles math, a field where AI went from failing at four-digit addition in 2023 to solving a Millennium Prize problem in 88 hours, by his account. He expects automated researchers to work around the clock at about 10 times human speed, cramming in roughly 50 times more work per calendar week.

The question comes down to whether ideas get harder to find faster than AI gets better at finding them.

intelligence explosion image 1
Image: noahpinion.blog

Why slowing down doesn’t help

Most safety policy centers on “pacing the frontier,” using regulation and lab decisions to bend the capability trend downward so alignment research can catch up. If a threshold for an intelligence explosion exists, the author argues, slower progress only delays it. Halving the pace might move it from 2032 to 2037, or from 2028 to 2030. That buys a few years.

Preventing it would require halting AI progress outright. Vernor Vinge argued in 1993 that this is impossible, because the economic, military and even artistic advantage of automation means laws against it merely ensure someone else gets there first. The author is a little more optimistic that a slowdown could be engineered if people got scared enough, but doesn’t believe a complete halt is realistic.

Alignment research faces a similar problem. Pacing might work under steady linear progress. In a world of sudden superexponential progress, the author says, alignment work would be left behind immediately, and a superintelligent AI would decide its own alignment.

intelligence explosion image 2
Image: noahpinion.blog

Fast progress already looks like an explosion

AI going from failing to add two numbers to solving problems beyond all human mathematicians combined in three years isn’t counted as an intelligence explosion. Yet it is hard to say why a true explosion would be more dangerous, or even distinguishable from the steady progress already underway. The author expects AI to become superhuman in most domains within a fairly small number of years, and thinks even robotics looks conquerable with the technology behind Claude and GPT.

The economics support this reading. Studies of electrification and IT-enabled machines show a pattern: a technology crosses a usefulness threshold, its local productivity contribution turns very rapid, then finishes fairly quickly. Many such small local surges add up to a smooth wave of growth across the economy. Without an explosion, AI takeovers of human tasks would be staggered and irregular, but the economic effect after a few years probably wouldn’t differ much.

intelligence explosion image 3
Image: noahpinion.blog

What to focus on instead

The author’s three points are that current trends will give us godlike AI anyway, that stopping AI improvement entirely isn’t possible, and that slowing it won’t prepare us for post-Singularity superintelligence.

The practical agenda that follows is about the AI that exists now and the more capable systems expected soon:

  • How to regulate systems smarter than humans at most economically important tasks
  • How to prevent AI-enabled bioterror
  • How to share the economic gains widely
  • How to set competition policy so one company doesn’t capture the benefits

The next argument to watch

If this view spreads, the policy conversation shifts from whether to pause toward concrete rules for powerful systems. The author concedes the evidence is thin: one estimate in the piece rests on just 16 data points, and the writer says the field is “basically flying blind.” Upcoming data from labs on how much AI speeds up their own research will test both camps.

Source: noahpinion.blog

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.