AI-assisted code recently produced a proof of the Riemann hypothesis, roughly 17 million lines of it, and the result has been certified correct even though no person yet understands how it works. That has pushed one systems researcher to ask a blunt question about software complexity: did humans ever manage it well?
The answer from their analysis is mostly no. Decades of human-written code are turning out to be full of bugs, and the problem is getting worse as AI speeds up attackers.
Software complexity was already out of control
There has always been a gap between what computer science understands about computation and how fast software has been pushed into every corner of society. Software now runs much of daily life, which is how odd situations arise, like supermarket fridges needing serious security work. Mathematician Tim Gowers has joined the Pause AI march, a sign of how uneasy some experts have become.
Software complexity was manageable when attackers and defenders were roughly evenly matched. Recent correlated outages and disclosed flaws in Java, PHP and Linux suggest that balance is slipping, and the author expects the pain to accelerate in 2027. They see two ways out, then add a third conclusion about society.

Option one: derive software from specifications
In April 2026 the author doubted that anyone would formally specify every layer of hardware and software. Frontier model progress has changed their mind. Translating the huge body of legacy code now looks plausible, provided the industry first races to write formal specifications for its services and derives new software from them.
That means rebuilding operating system stacks from the ground up and even rethinking instruction sets. Many awkward interfaces exist only because software layers grew so human teams could work on separate parts. Unikernels, which link a library operating system into each application, and shared-memory interconnects are composable enough to let agents specialize a whole stack at link time. Fewer interfaces make the specification job smaller.
Specifications also take humans out of the security patch loop. Attackers are limited only by their appetite for breaking in, while defenders wait for maintainers to triage reports. A fix re-derived from a spec and checked by machine could arrive faster and more reliably. People would keep the job of writing interfaces with an AI, iteratively.
Option two: build in diversity
Software tends toward monoculture because of reuse, so a single bug can hit vast numbers of hosts. Agents could instead generate deliberate variation, using something like genetic mechanisms, so services aren’t all infected at once. This pairs well with formal specs, since a good spec allows many valid implementations. The Internet already leans this way through Postel’s Law, which asks hosts to be liberal in what they accept and strict in what they send.
There’s a catch. If a handful of gated frontier models write all the software, the industry trades one risk for an AI-controlled one. Diversity has to cover models and tooling as well as code. Internet RFCs haven’t splintered much, so specs might stay singular while implementations vary.
Why better models won’t tame software complexity
The author doesn’t think stronger models and better tooling will let software engineering catch up by themselves, since they mostly produce piles of sloppy code. The two approaches need to work together. Specs let hosts reject malformed input instead of accepting whatever arrives, and diversity limits the damage from whatever the specs miss.
The biggest disruption may hit social coding. Open source has run for decades on human review and earned trust, and every community the author belongs to is now struggling with AI disclosure.
What humans still have to do
People still decide what goes in the specs, as they have with RFCs for decades. Open research questions remain:
- How to debug a system when no two nodes run the same code
- How to force simplicity on systems that agents maintain
- How to assign blame when things go wrong
Those questions will shape how the next generation of software is built, and who answers for its failures.
Source: Anil Madhavapeddy