Coding agents have written custom megakernel inference engines for eight open models, and the engines beat vLLM and SGLang on single-sequence decoding. Speedups ran from 1.13x to 1.77x with the original data flow intact, and reached 2.05x when the model’s storage was allowed to change. The models range from 8 billion to 594 billion parameters.
One kernel instead of many
Megakernel inference engines collapse the whole decode loop into a single launch.
General engines such as vLLM and SGLang are built to run any model. Every decoding step is assembled from a shared library of kernels, with a separate one for attention, for matrix multiplication and for normalization. A CUDA graph launches them back to back, and the CPU does the scheduling each time.
The megakernel goes to the other extreme. An agent writes one GPU program for a single checkpoint on a single GPU type, and that program handles all operations in every step, along with the loop across tokens. With everything in one kernel, the agent controls ordering freely. It can load upcoming layer weights during current computation, merge operations so intermediate results stay out of memory, and let idle thread blocks begin the following phase ahead of time. Splitting the work across launches blocks these tricks.
The price explains why megakernels stayed rare. They’re hard to maintain and debug, tied to one model and one GPU, and they need retuning when batch size or context length changes. Until now they existed as hand-written versions for a few models, such as Hazy Research’s for Llama-1B and Cohere’s for North Mini Code, or as compiler output for a set of supported architectures. The team’s argument is that coding agents have removed the cost.
Two tracks, eight models
Track A leaves the reference data flow alone. Weights stay as the checkpoint stores them, and activations and the KV cache use the same layouts as vLLM and SGLang. The only thing that changes is how the GPU schedules the work. Track B requires only that the function stays identical in exact arithmetic, so the agent may change how the model is stored and what’s computed at load time. Lossy quantization and pruning don’t qualify.
Tests ran on H100 and B200 GPUs, covering dense and mixture-of-experts models. The standard decode test feeds a short 64-token prompt, generates 1,024 tokens with greedy decoding, and runs at batch size 1 or 8. Each system sat behind an HTTP server on identical hardware.
With one sequence, track A beat the best general engine by 1.13x to 1.77x across all eight models. Eight sequences at once narrowed the lead on six models, to as little as 1.02x for North Mini Code and 1.08x for gpt-oss-120b. For North Mini Code the strongest rival is Cohere’s own megakernel, so the 1.99x and 1.37x results quoted there compare against vLLM.
Single-sequence decoding is limited by memory bandwidth. Take Qwen3-8B on an H100: streaming its 15.1 GB of BF16 weights through memory once at the 3.35 TB/s peak needs 4.52 ms. The track A engine produces a token in 5.13 ms, which is 88% of the theoretical best.
What track B found
On Qwen3-8B the agent spotted waste in the BF16 format: it allots 8 bits to the exponent, but values within a trained matrix row cover only a handful of binades. So it stored every weight in 12 bits with no loss and expanded them inside registers just ahead of the tensor cores. Weight reads per token fell by 25%. Two more techniques showed up repeatedly. One is a certified low-precision approximation of the LM head that narrows down candidates for the next token. The other is a lookup table per token for layer 0, since that layer sees only the token’s embedding. Where checkpoints are already compressed, like the 4-bit experts in Inkling-2-Small and gpt-oss, track B gained little.
Serving real traffic
Benchmarks at batch size 1 flatter any engine, so the team also tested megakernel inference engines under mixed load.
A server has to admit new requests while others decode. For Qwen3-8B and Inkling-2-Small, a follow-up run pushed the scheduler itself onto the GPU. That brought continuous batching, chunked prefill, a paged KV cache and prefix caching, all running within the kernel.
On a production trace with Inkling-2-Small on four B200 GPUs at one to eight requests per second, 99% to 100% of requests met their service levels, against 82% to 85% for SGLang. The longest stall was 0.08 to 0.15 seconds, against 2.3 to 4.4 seconds.
How the agents were kept honest
The team built a framework where agents design megakernels on their own, with Claude Opus 5.5 and Fable 5.2 doing the work. The central piece is a test suite that the agent has no way to edit. An engine is timed only after it matches a high-precision reference numerically, on unseen inputs and while running, and after an independent audit.
The megakernel inference engines, an OpenAI-compatible server, the test suite and tools for reproducing the work on other models are released as megakernel-zoo. For anyone running a single model at scale on a fixed GPU, the open question is how often a per-checkpoint engine is worth the retuning when batch size or context length shifts.
Source: dangxingyu.github.io