In a Gaudi2 vs H100 test on long-context inference, a single Intel Gaudi2 accelerator ended up 60% faster at decoding than Nvidia’s H100 80GB PCIe card. The startup Standard Kernel got there with software changes alone, and says the tuned Gaudi2 costs far less to rent.
The result is a narrow one, measured on one workload and one model. Even so, it revives a chip line Intel had effectively written off.
The numbers behind the Gaudi2 vs H100 result
Standard Kernel ran Qwen3.5-27B in FP8 on both cards, with 16 concurrent requests. Each request carried 70,000 input tokens, and 95% of that prefix was shared between requests. That pattern resembles coding agents that keep re-reading long conversation histories or large codebases.
Against its own untouched baseline, the company raised Gaudi2’s cached-prefill throughput by 3.21x and its decode throughput by 2.84x. Compared with the H100, the tuned Gaudi2 came out 7.92% ahead on cached prefill and 60.35% ahead on decode.
Both systems ran vLLM 0.32.0. The Gaudi2 side used the vllm-gaudi plugin dated October 26, 2026, with and without Standard Kernel’s changes.
Why the price gap matters more than the speed gap
Standard Kernel found Gaudi2 for rent at $0.60 an hour, against $3.50 an hour for an H100. That is nearly a sixfold difference before any performance is counted.
Put the two together and the tuned Gaudi2 produces 9.4 times more decoded tokens per dollar. For newly computed cached-prefill tokens, the advantage is 6.3 times per dollar. For anyone paying by the hour to serve long prompts, that is a large swing.
Why Gaudi never took off
Intel bought Habana Labs in 2019 for roughly $2 billion, hoping to take on Nvidia in AI accelerators. On paper, Gaudi2 had the specs to compete. It carries 96 GB of memory against the H100’s 80 GB, though the H100 has more bandwidth, at 3.35 TB/s versus 2.45 TB/s.
Adoption stayed limited anyway. In January 2025, Intel canceled Falcon Shores, the chip meant to carry the Gaudi line forward.
Standard Kernel blames software. Nvidia has spent years building compilers, kernels, inference engines and model-specific optimizations. Rival hardware often supports the same operations but runs slow or incomplete implementations of them. A weak benchmark can therefore say more about missing software than about the silicon.
What the optimizations changed
The company describes gains from across the inference stack rather than one trick. The main changes it lists:
- Finer 128-token buckets for prefill, which cut wasted padding and let more requests fit in the scheduling budget without being split.
- Faster prefill attention that handles cached prefixes separately from causal suffixes, with batched matrix operations and larger cache-aligned tiles.
- Smaller blocked matrix inversions in place of expensive 128×128 ones for the model’s GDN layers, backed by a custom TPC kernel.
- FP16 for intermediate tensors, with numerically sensitive operations kept in FP32.
- Gathering shared key/value data once per decode step instead of once per request.
- Keeping recurrent state in dense buffers across decode steps, and fusing convolution, activation, Q/K/V extraction and normalization into a single TPC kernel.
Several of these target memory traffic. On long shared contexts, moving data costs more than arithmetic, so trimming repeated reads pays off quickly.
What to keep in mind
This is a vendor’s own benchmark of its own optimizations, on a single model and a single request pattern. A different model, shorter prompts or lower prefix sharing could narrow or erase the gap. The H100 was also tested as the PCIe 80GB card, not a faster variant.
The rental prices are what Standard Kernel found available, and cloud rates for either chip move around. The 9.4x figure depends on both numbers holding.
What happens next
Standard Kernel is building a system that can try changes across the inference stack on its own, benchmark them on real hardware, check correctness and keep what it learns. It argues that AI inference will become heterogeneous, with software deciding which accelerators are worth buying.
Whether other teams can repeat the Gaudi2 vs H100 result on different models is the open question. If they can, older accelerators sitting in data centers may be worth more than their spec sheets suggest.
Source: standardkernel.com