A team working on DeepSeek-V4.1-Flash serving on Nvidia H20-3e GPUs cut mean end-to-end latency by 65.5% and raised output throughput 2.91 times compared with a plain tensor-parallel setup. The gains come from a disaggregated design that splits prefill and decode across separate GPUs, plus a series of kernel and scheduling changes made through SGLang.
Mean latency fell from 13.63 seconds to 4.70 seconds. Output throughput climbed from 847 to 2,465 tokens per second.
The setup
The DeepSeek-V4.1-Flash serving tests were designed so that each change could be measured on its own, with adjacent configurations compared step by step.
The test used an eight-GPU host with four GPUs for prefill and four for decode. A router sends each request through both stages, and the KV cache moves between them over NVLink using Mooncake. The baseline ran TP4 attention and EP4 mixture-of-experts on both workers, with eager BF16 Torch scoring in prefill.
Every run used the same checkpoint, Humming W4A8 MoE kernels, a DSpark block size of 5 and 8,192-token prefill chunks. Each run completed 256 requests and produced 128,000 output tokens.

Where the speedup came from
Most of the improvement arrived before the first token. Mean time to first token dropped from 12.04 seconds to 1.89 seconds. Mean time per output token rose from 3.20 to 5.62 milliseconds, a trade the team accepted because requests spend far less time waiting.
The prefill queue shows the effect. Its median length fell from 15 requests at baseline to 9 after the first change, then to 4 and 3 with later steps, and to zero once tensor-parallel MoE was added. The decode queue stayed at zero throughout.

The seven changes, ranked by impact
- Prefill context parallelism with decode data parallelism cut mean latency 44.64% and lifted throughput 81.62%. It was the biggest single gain.
- FP8 DeepGEMM scoring in the prefill Indexer cut latency another 29.85% and added 43.11% throughput.
- Breakable CUDA graphs for prefill trimmed latency 3.66%.
- Tensor-parallel MoE cut it a further 5.67%.
- Larger Indexer score tiles changed latency by 0.79%, within run-to-run variation.
- Fusing prefill selection and writeback reduced time to first token by 1.92%.
- Reusing paged Indexer K tiles during verification trimmed latency 1.59%.
Why context parallelism helps prefill
Prefill processes many token rows from one long request, so interleaved context parallelism spreads those rows across the four GPUs. Decode is different: it handles many requests with few query rows each, so data-parallel attention gives each rank its own requests.
The FP8 scorer needed care. Queries and keys are converted to E4M3, and the team checked what that did to results. On a 64K prompt, none of the 16,774,528 sampled key values needed clamping, and the BF16 and DeepGEMM scorers agreed on 99.08% of selected top-k entries.
Why tensor-parallel MoE won out
With expert parallelism, a skewed router sends more work to a few experts, and the other ranks wait. Profiling caught one rank idling about 100 microseconds at an all-reduce. Tensor-parallel MoE splits each expert across all four ranks, so uneven routing no longer loads one GPU more than another. In prefill, two profiled batches took 771 milliseconds with expert parallelism and 657 milliseconds with the tensor-parallel version, even though the expert matrix multiplies were slower.
Colocated serving
Not every deployment has eight GPUs to spare, so the team also measured DeepSeek-V4.1-Flash serving on a single worker.
That worker had four GPUs. Against the BF16 TP4 baseline, mean latency fell from 18.62 to 7.96 seconds, and throughput rose from 642 to 1,483 tokens per second. Adding decode data parallelism cut latency another 7.82%.
The colocated setup produced more tokens per GPU, 371 against 308. The eight-GPU disaggregated setup had lower latency at 4.70 seconds versus 7.96 seconds, and higher total throughput.
What happens next
Several of the changes are still under upstream review in SGLang. The team plans to test expert parallelism with all-to-all communication and expert load balancing, aiming to keep the faster matrix multiplies while spreading work evenly. The public branch lists the exact commit and environment for anyone who wants to reproduce the numbers on their own H20 hardware.
Source: lmsys.org