The AI superintelligence debate got louder this summer when employees at a leading lab started openly discussing human extinction. A computer scientist who taught a doctoral seminar at Georgetown University now says one question is missing from it: is superintelligence possible from an engineering standpoint? The answer, they argue, is probably not through the route labs describe, and chasing it could still do harm.
What sparked the argument
Anthropic researcher Jacob Coxon resigned and warned that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied that the company earnestly believes AI could kill all humans. CEO Dario Amodei later told CNN, “I agree with Jacob much more than I disagree with him.”
Those worries rest on an idea that has circulated in futurist circles for decades. Once engineers build an AI that can improve itself, the theory goes, it triggers an accelerating loop called recursive self-improvement, or RSI. The result would be artificial superintelligence, a system whose abilities far exceed ours and that could escape our control. That is a higher bar than artificial general intelligence, which means matching humans across a broad range of cognitive tasks.
The AI superintelligence debate has split into camps. Some say ASI talk is marketing, or a way to win regulation that shields incumbents. Others debate scenarios such as AI-engineered viruses. Still others hold that alignment is feasible and the field should hurry on.
An old idea, now an engineering project
Statistician I.J. Good floated an “intelligence explosion” in a 1965 paper. The Extropians, a Bay Area futurist group, picked it up in the 1990s, and Eliezer Yudkowsky and Nick Bostrom carried it further.
It has since moved from thought experiment to lab practice. In a June post titled “When AI Builds Itself,” Anthropic said LLM agents are speeding up writing, debugging and optimizing code, and automating parts of research like running experiments. Its internal data, it said, shows AI is already accelerating AI development. In September, OpenAI announced a goal of an automated researcher by March 2028.
The data obstacle
The Georgetown seminar’s first objection is that RSI remains speculative, and data is the reason. Through 2023, big gains came from scale. GPT-1 had 117 million parameters, GPT-2 had 1.5 billion, GPT-3 had 175 billion, and GPT-4 likely exceeds a trillion. After GPT-4, scaling brought diminishing returns, and engineers turned to post-training with reinforcement learning, which rewards a model for successful responses.
That approach needs many examples of inputs paired with correct outputs. Such data is easy to produce in structured fields like programming and mathematics, which helps explain rapid progress there. Areas without it have advanced less, which may be why AI can tackle advanced theorems yet struggle to empty an email inbox.
French researcher Franรงois Chollet has asked whether the jagged frontier is mainly math and code, which can be pushed arbitrarily far with reinforcement learning, while everything else plateaus because it’s bottlenecked by human-generated data. The author’s view is that coding agents and automated researchers won’t remove that limit. Claims that RSI is imminent also understate the scientific effort behind major AI advances.
The timeline helps show how far the idea has travelled. Turing described a universal computing machine in 1936. Good’s 1965 paper followed, then backpropagation in 1986, and Vernor Vinge’s 1993 argument that faster thinking alone isn’t enough: a faster-thinking dog still couldn’t play chess. Deep learning breakthroughs in the 2010s brought larger models, more data and more computing power. By the 2020s, Amodei and other AI leaders were naming superintelligence as a goal, with RSI proposed as the path.
The seminar’s reading list, published as an annotated bibliography, traces the same technical and cultural roots. In 2026, AI systems are already accelerating AI development, and labs are building automated researchers. Whether that leads to an explosion is the point in dispute.
Three futures
The piece lays out three scenarios:
- An intelligence explosion in which AI surpasses humans and may escape control.
- Danger without superintelligence, where more autonomous systems cause harm through misuse, poor design or loss of control.
- A useful future of powerful specialized tools whose behavior is constrained and predictable.
What to watch next
For the AI superintelligence debate, the practical question is where progress comes from. The author’s deeper worry is that even if the explosion never arrives, pursuing it can cause real harm. The test is whether automated researchers show gains outside coding and mathematics. If they don’t, the argument against imminent RSI gets stronger.
Source: IEEE Spectrum