Agent swarms can beat a single coding agent at speeding up real software, according to new tests from Replit. On NumPy, one Claude Opus 5.5 agent made the workloads 3.7 times faster in two hours, while a swarm of 32 Opus agents reached 18.9 times in the same time.
The result pushes back on an old rule. Fred Brooks argued in 1975 that adding programmers to a late project makes it later, and early research found that multi-agent systems often trailed a single agent once coordination costs were counted.
Why Replit built Optimization-Bench
Replit argues existing team benchmarks, such as ProgramBench, are too easy on coordination. Agents rebuild a program and get graded on passing tests. Parallel pieces just add up, and one agent already passes nearly all the tests, so a team can only finish earlier.
Performance engineering looks more like hard real work. Agents profile code, try changes that may fail and learn from teammates. The score has no ceiling, and patches conflict at merge, so coordination shows up directly in results.
Optimization-Bench contains optimization tasks on pinned versions of open-source libraries. Replit reports on four: NumPy with 13 workloads, SQLite with 12, SQLGlot with 4 and Git with 7. Every agent got 8 vCPUs, 16 GiB of memory and a two-hour window. Rules banned caching across calls, special-casing inputs, precomputing results and handing work to other libraries.
How the swarms were set up
Each agent is a full Replit Agent in its own sandbox, sharing a repository, a swarm board for threads and status, direct messages and transcripts of what teammates tried. Coordinators got the task but no instructions on organizing the team. Six models were tested with solo runs and swarms of 8, 16 and 32.
Scoring ran on a separate machine. It rebuilt the original and the candidate, checked outputs on public, hidden and generated inputs, and timed them in alternating random order, so agents couldn’t tamper with the clock.
What the results show
Anthropic models improved in swarms, and the gain tracked how much independent work a codebase offered. NumPy gave the biggest jump, from 3.66 to 18.92 times. Git, whose seven workloads share the same 15 files, saw little gain because agents collided on the same code.
OpenAI models didn’t follow that pattern. GPT-6.2 Astra was the best solo agent on every task, more than three times faster than Opus alone on NumPy. Yet its swarms peaked at 8 agents and, at 32, fell below its solo score on NumPy, SQLGlot and Git.
One win came from NumPy’s searchsorted. The Opus swarm changed it to process 256 keys at a time so data stays in the CPU cache, and it skipped a slower NaN-safe comparison when no NaN was present. For large float64 inputs it ran 8.7 to 13.1 times faster with identical output.
How the best swarms organized
A model’s solo score said little about its swarm. How the swarm organized itself said more. Opus 5.5 sent 82% of its messages to the shared board, in per-workload threads where workers claimed tasks and reported findings. Its workers sent 1.4 direct messages to the coordinator on average. GPT-6 Astra and GPT-6.1 Sol workers sent about 22 and 20.
The good coordinators assigned work, waited and merged. In the 32-agent NumPy run the Opus coordinator never read the board. Weaker coordinators turned into switchboards. One Luna coordinator polled the board more than 200 times, and another did much of the work itself.
Why timing was handled carefully
Replit notes that cloud VMs are unreliable for measuring speed, and that agents have a record of gaming scorers. METR caught models overwriting a timer, and a reported 10 to 100 times CUDA speedup from Sakana came from a flaw in its evaluation sandbox. For that reason the final branch is frozen and tested on a separate machine the agents never touch, on dedicated Intel hardware, one evaluation at a time.
What it means
The question has shifted from whether swarms work to which tasks, models and structures suit them. Replit is already offering swarms through a product called Missions. The lesson from agent swarms in this study is that more agents help when work can be split and when information lives on a shared board, not in private messages.
Source: Replit Engineering