A language model can load onto a GPU and still leave too little room for the requests a service has to handle. KV cache sizing is the step that closes that gap: it turns prompt length, answer length and the number of simultaneous generations into a memory budget.
The fix starts with keeping two questions apart. Does the model fit, and how much serving capacity is left?
What the cache holds
A key-value cache stores the attention keys and values for tokens a transformer has already processed. During prefill, the model reads the prompt and builds the first cache. During decode, it generates new tokens, reusing cached state and adding to it. Each attention layer keeps its own.
Caching removes repeated work. It does not remove the cost of reading a longer history, because full attention still looks at everything retained.
Weights and cache also live differently. Requests on the same loaded model share the weights, but each request has its own prompt and continuation, so its attention state differs. A new replica needs another copy of the weights. Admitting more requests to an existing replica mostly adds cache.

The formula for KV cache sizing
For a standard decoder with identical full-attention layers, bytes per retained token are:
2 x layers x KV heads x head dimension x bytes per element
The 2 covers keys and values. BF16 and FP16 each use two bytes per element. Multiply the result by the total retained tokens across all active sequences. Count the full formatted input, including system instructions, conversation history, retrieved documents and tool results, then add room for the output still to come.
Architecture changes the answer. Grouped-query attention lets several query heads share key/value heads, and storage follows the KV head count, which can be much smaller.
A worked example with Mistral-7B
Mistral-7B-Instruct-v0.3 has 32 layers, 8 KV heads and 32 query heads, with a hidden size of 4,096, so each head is 128 wide. The cost per token is 2 x 32 x 8 x 128 x 2, or 131,072 bytes (128 KiB).
The example assumes one NVIDIA L40S with 48 GB of advertised memory, one replica with BF16 weights and cache, and a 16 GiB KV budget after allowing for weights, runtime memory and headroom. That budget is an assumption, not a measurement.
A request with a 6,144-token prompt and a 2,048-token output allowance needs 8,192 tokens, or 1 GiB. Raise the planned length from 8,192 to 32,768 tokens and the memory-only ceiling falls from 16 concurrent sequences to four, with no change to the weights.
Mixed traffic needs an admission budget
Real traffic mixes lengths, so the individual needs add up. Two sequences planned for 32,768 tokens use 8 GiB. Eight more at 8,192 tokens use another 8 GiB. Ten sequences exhaust the budget, even though a short-request-only estimate suggested 16 would fit.
Output belongs in admission control, the decision to start, queue or reject work. A server may admit many short prompts at first, then find their answers growing into the same remaining blocks. A workload contract needs both a concurrency limit and a token-growth allowance.
Reserved memory is not active pressure
A serving process can reserve a large cache pool before requests fill it, so high GPU memory use alone says little. The pool has three kinds of blocks:
- Active blocks, which running sequences still need.
- Reusable, unreferenced blocks, which keep earlier prefixes for possible reuse and can be evicted without interrupting anything.
- Free blocks, which are available immediately.
Evicting an idle prefix costs a possible future saving. Preempting active work pauses progress and may require recomputing or moving state. A cache hit also needs a compatible model state and a matching token prefix, with any tenant or security boundaries enforced. Similar wording is not enough.
What to do with the numbers
The figures above are payload-only upper bounds. They exclude allocator overhead and metadata, and they say nothing about latency or compute. Paged allocation rounds each sequence to block boundaries, and shared prefixes can cut duplication where the runtime shares physical blocks.
Before sizing a deployment, write down the maximum prompt, the output allowance and the number of sequences that must stay resident. Then do the KV cache sizing for your own model, and keep prefix retention as a separate budget.
Source: aranya.tech