RightNow AI says its FP4 MoE kernels for Nvidia’s Blackwell chips produce exactly the same numbers whether a prompt goes in one position at a time or 64 at once. The Y Combinator-backed lab tested the claim on Inkling 2 Small, a 594-billion-parameter mixture-of-experts model trained in NVFP4 and spread over eight B300 GPUs.
In the test, all 599 positions it checked gave identical logits at both settings. Compared with vLLM on the same hardware, next-token probabilities differed by 0.025 nats on average.
Why bit-identical output matters
Inference engines usually get close, not identical. Which other requests share a step changes how partial sums are grouped, and floating point addition gives different totals depending on grouping. So when a vendor calls an optimization lossless, it normally means accuracy moved only within noise.
RightNow works to a tougher standard known as batch invariance. Under it, a row’s answer can’t change with whatever else is in the batch. The lab holds that line through both low-precision tensor core formats, the expert split over eight GPUs and each data exchange between them.
The lab says its own AI kernel engine produced every kernel. In that system, coding agents write CUDA, run the tests and either keep or roll back each change. A kernel survived only if it reproduced its predecessor exactly, or, where the math was altered deliberately, stayed batch invariant inside a stated tolerance.
How the team kept results stable
- Sums always run in the same chunks and balanced trees, and totals across GPUs are combined in rank order in FP32.
- Compiler contraction is switched off, so the only fused multiply-adds are ones the engineers wrote on purpose.
- Every property comes with a sabotaged twin, and the tests have to catch it.
Kernel 1: splitting FP4 experts across GPUs
The model’s first 40 layers keep their experts in NVFP4 and the last eight use MXFP8. The serving engines the team looked at keep each expert whole on one GPU. They do this because cutting an expert’s columns apart hides part of the row that the next matrix multiply must scan for its maximum.
RightNow cut them anyway. Each GPU takes an eighth of every expert and the GPUs swap their row maximums. Taking a maximum gives the same answer in any order, so nothing is lost. Results matched across every ordering of the eight ranks the team tried.
A second variant swaps the activations themselves and divides one weight matrix by output rows. That cuts traffic by a factor of 3.2 per FP4 layer, from 129 KB to 40 KB per GPU at a single row.
Kernel 2: the matrix multiply, rebuilt for single tokens
With a single token, each GPU must stream 11.9 MB of FP4 data per layer through the first matrix multiply. The original kernel put only 40 thread blocks to work on a chip with 148 multiprocessors and managed 114 GB/s. The new one divides the work along the K dimension and overlaps memory loads with computation, keeping the operation order unchanged.
Memory throughput rises to 583 GB/s for one token. With eight tokens it reaches 1.39 TB/s for NVFP4 and 1.84 TB/s for MXFP8, still short of the 6.85 TB/s the B300’s memory can deliver. Compared with the original, none of 512 test rows differed.
Kernel 3: a one-round depth mix
Before each attention and expert block, Inkling blends in the outputs of earlier layers. By storing a Gram matrix of those outputs, the team needs one reduction per blend rather than two, which means two rounds per layer instead of four. This kernel keeps batch invariance but not exact bit equality with the direct method, since the norm differs by about 1.6e-7 in relative terms.
The team also tried merging the expert router into that reduction. On two real layers it altered the chosen top eight experts for 1.1% and 1.6% of tokens, so it was dropped. Shifting routing would break the lossless goal.
Where the speed stands
At present the engine runs each decode step as separate launches under CUDA graphs. One row with no speculative draft takes roughly 16 ms on eight B300s. Profiling shows 5.5 ms going to the first kernel’s w13 multiply, which the new version shrinks about five times in isolation but hasn’t been merged into the engine, and 3.3 ms to GPU-to-GPU exchanges.
Next the lab plans a single long-running program on each GPU, holding every layer and the exchanges. It projects about 1,900 tokens a second per user, in a range of 1,630 to 2,160, based on an assumed 45 active experts per layer that it hasn’t measured. vLLM, using the same draft, delivers 666 tokens a second per user on the lab’s own eight-GPU machine. A measured figure is still to come.
Source: RightNow AI