SGLang llm-d Support Cuts GLM-5.2 Latency 8.7x

Diagram of SGLang llm-d architecture with tiered KV cache on Kubernetes

The llm-d project now runs SGLang as a first-class, Kubernetes-native engine in its GLM-5.2 reference deployment. The SGLang llm-d work came from contributors on both sides, and it sits next to the existing vLLM path. Benchmarks on NVIDIA H200 GPUs show how much the cache setup matters for long agent sessions.

One headline result: on a 16-GPU layout, moving evicted cache blocks to host RAM cut median time to first token (TTFT) at 128 concurrent sessions from 174.2 seconds to 20.1 seconds.

One platform, two engines

llm-d is a Kubernetes layer for serving large models. It handles scheduling, routing and prefill/decode splitting, and it keeps those jobs out of the model server itself. That separation is why a second engine did not need a second stack. SGLang runs on the same DisaggregatedSet orchestration, the same per-rank endpoint discovery and the same prefix-aware routing that vLLM uses. Operators swap an engine-specific overlay and a router configuration, and the rest stays put.

DisaggregatedSet runs multi-node prefill and decode pools as native Kubernetes workloads. It brings gang scheduling, rolling updates and Dynamic Resource Allocation for GPUs and RDMA network cards. Going from a 16-GPU single replica to 32 or 64 GPUs uses the same declarative setup.

Engine-agnostic llm-d architecture with Kubernetes-native SGLang Wide-EP and Shared L3
Image: llm-d.ai

Making every DP rank its own endpoint

GLM-5.2 runs with wide expert parallelism. Experts are shared across GPUs, but each attention data-parallel (DP) rank keeps its own KV cache. llm-d routes at the rank level: it tracks which prefixes each rank holds and how many tokens are in flight. So one endpoint per node would not work.

SGLang could already serve GLM-5.2 on its own, but each rank needed its own port, its own prefill/decode pairing handshake and its own health and observability signals. Contributors added per-rank Rust endpoints and replicated the pairing state inside SGLang. The router can now pick the prefill rank with the warm prefix and the least-loaded decode rank, even across multi-node pools.

Extending GLM-5.2 KV-cache capacity from GPU HBM to host RAM and shared storage
Image: llm-d.ai

Three cache tiers

With routing sorted, the bottleneck for agent workloads is KV-cache capacity. The tests used SGLang’s HiCache on the prefill pool, on 8-GPU H200 nodes on GKE A3 Ultra with ConnectX-7 RoCE and a 9,000 GiB Google Cloud Managed Lustre volume.

  • L1, GPU memory: 261K tokens per GPU, or 2.09M tokens per 8-GPU prefill pod. That holds roughly 20 to 30 repository-scale sessions.
  • L2, host RAM: pinned memory per rank, reached over PCIe Gen5 DMA. A single prefill pod grows to 17.5M tokens across L1 and L2.
  • L3, shared storage: SGLang’s file backend on a ReadWriteMany volume, holding about 163M tokens. GLM-5.2’s KV format is identical across DP ranks, so all ranks and replicas share one deduplicated pool.
L2 host RAM eliminates the L1 thrashing cliff on a single 16-GPU replica
Image: llm-d.ai

What the numbers show

The single-replica test used the 900-second AIPerf agentic benchmark with a Claude Code trace corpus and a 142K context cap, at concurrency from 16 to 128. Past 32 sessions, active work exceeded the 2.09M-token GPU budget and sessions evicted each other. Cache hit rate fell to 6.0%.

Host RAM offloading held cache reuse at 85% to 95% across the sweep. At concurrency 128 it raised output 3.94x, from 319 to 1,257 tokens per second, and cut median TTFT 8.7x.

Shared L3 did not help on a single warm pod. Its write-through backups and synchronous prefetches added 14% to 33% to median TTFT and trimmed output by about 9% at concurrency 128.

The multi-replica test used 32 H200 GPUs: two prefill pods and a two-node, 16-GPU decode pool. At concurrency 128, median TTFT fell 5.9x to 3.43 seconds at 1,903 tokens per second. Shared L3 added overhead in steady state, raising median TTFT 26% at concurrency 64 and 66% at 128. It paid off when things broke.

  • In a live failover test, the team deleted a prefill pod at t = 95 s with 24 active sessions of about 98K tokens each. The survivor restored 99.94% of the moved prefix tokens from storage instead of recomputing them. Median TTFT fell 25%, from 38.4 s to 28.8 s.
  • In a cold-restart test with 16 sessions of about 92K tokens, recovery TTFT fell 57%, from 38.8 s to 16.8 s.

Which setup to run

For steady single-replica pods, L1 plus L2 host RAM is the better choice. Add Shared L3 if you expect Spot preemptions, upgrades or multi-replica failover. The catch is the extra latency L3 adds when nothing fails.

Deploying takes two steps. Install the standalone inference router with the SGLang values file, then apply a deployment overlay from the GLM-5.2 guide. The 32-GPU p2w1d1w2 overlay uses L1 plus L2. The 16-GPU layout with Shared L3 needs a ReadWriteMany volume named llm-d-kv-cache-storage.

Exact cache awareness is the next target

The team plans to connect SGLang’s Rust preprocessing layer, sglang-processor, to the llm-d router. Prompts would be rendered and tokenized once at ingress, which avoids token drift between router and worker. Paired with per-rank metrics and KV-cache event streams across L1, L2 and L3, routing would move from approximate prefix tracking to exact, block-level cache awareness.

Source: llm-d.ai

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.