Gemini 4 Neon passed 86 of 92 runs on VulcanBench Frontier v4. Every task fell at the three higher settings, and Low managed 17 of 23. The logs behind those scores show an unusual working style: Neon spends most of a run studying the problem, then fixes it in one edit.
The task
Each task ships a small repository: a buggy module meant to copy a discontinued program, a written spec that may not match what that program does, and the program’s own macOS and Linux binaries. Hidden tests diff the fixed module’s output against the binary on unseen inputs, so the binary settles every dispute.
How Neon works through it
Roughly three quarters of a run is over before Neon edits anything, at every setting. It opens by listing files and reading four or five. At Low and Medium it often feeds the old program a few typed commands first (16 of 23 runs each), a habit that shrinks to 4 of 23 at Extra-high.
Disassembly follows, usually around tool call nine or ten and within a minute. From High up, 18 of 23 runs inspect both builds. The longest reasoning stretch comes after that, then experiments against the original program, the rewrite and a randomized comparison. All 92 patches change only the module.
What extra effort buys
One task, blendcore, shows the spread. At Low it took 3.4 minutes, 15 tool calls and 6,926 thinking tokens. At Extra-high it took 10.3 minutes, 32 tool calls and 40,750 thinking tokens, yet the final rewrite was about the same size. The added time went into a second binary, a second look at the first and four experiment scripts, all before any edit.
Across the full set, thinking grows 4.4 times from Low to Extra-high (9.1k to 40.3k tokens per task), and runtime doubles from 5.1 to 10.0 minutes. Thinking climbs from 45% to 71% of output. Where Neon spends it changes: at Low, thinking splits roughly between reading, editing and testing. At Extra-high, 71% comes before it reads code or binaries, and testing falls to 15%.
Verification after the rewrite stays flat. Across 92 runs, Neon never planted deliberate bugs to test its checks, never ran the Linux reference in a container, never timed the rewrite and never tried a second Python version. Test runs per task go from 7.7 at Low to 3.3 at High and 4.6 at Extra-high.
Why Low failed six times
Every miss came at Low: cellarcore, hedgecore, lodgecore, matchcore, quotacore and tallycore, scoring 0.75 to 0.92. All six had read the machine code, so these weren’t blind guesses.
What differed was how the thinking was spread. Solved runs needed a median of 3 test runs and 2 edits, with 56% of thinking in a single burst. Failed ones took 7 test runs and 3.5 edits, and their largest burst was 25%. Total thinking was similar, about 6,900 tokens against 7,400. The failures look like patching by trial. At Medium, all six passed, and four of them used two to three times the Low thinking. Six cases can’t prove a cause, and harder tasks may simply produce both scattered thinking and failure.
Compared with Claude Fable 5.2
Claude Fable 5.2 passed all 115 of its runs on these tasks. It jumps to the machine code by its second to fifth call and edits sooner as effort rises, at 60% of the run at Low and 50% at Extra-high. Its added effort goes into proving the fix, through mutation testing and Linux container runs.
- Patch size: Fable 5.2 stays near 100 added lines at every level. Neon’s median climbs from 51 lines at Low to between 83 and 86 from High up.
- Comments: 12% to 19% of Fable’s added lines carry them, against 1% to 2% for Neon.
- Thinking: 9.1k tokens for Neon against 7.7k for Fable at Low, 15.6k against 7.1k at Medium, and about 40k each at Extra-high.
- Time at Extra-high: 10.0 minutes for Neon, 14.0 for Fable.
The models ran in different tool setups, and Neon’s reader opens one file per call, so minutes are a fairer comparison than call counts.
What to take from it
On this benchmark, more effort gets Neon a deeper grasp of the program but no extra evidence that the patch works. If you want a model to test its own output, Fable 5.2’s habits fit better here. Neon is weakest at Low, and none of this shows how either model does on other kinds of work.
Source: VulcanBench