LLM Output Caching Saves Only About 4% of Tokens

Output caching for LLMs sounds like an obvious win. Inference providers already cache model inputs so they don’t recompute a whole conversation on every message, so caching outputs seems like the next step. A closer look shows it would save very little.

The case for caching outputs

Models repeat themselves a lot. Coding agents are often asked to restate relevant code snippets while they explain a pull request. Long agent investigations produce evidence documents and then quote from them in replies. Automated AI research setups often have the agent repeat sample model outputs so a person can check them by hand.

Someone could read the files directly, but few do. Agents produce so much text that it’s easier to ask for the key points. That repetition wastes output tokens, which cost far more to produce than input tokens cost to ingest. Input tokens can be processed in parallel because of the attention mechanism, while output tokens must be generated one at a time.

How output caching would work

The fix would be to stop the model repeating itself. A coding model could be fine-tuned or prompted to write a file reference like “src/script.py:12-15” instead of pasting those lines back out.

More generally, a model could point to segments of its own conversation history instead of restating them. The software wrapped around the model could then expand each reference into the real content, so users would never notice the change.

Why the savings are tiny

The catch is that this only trims visible output. Models probably don’t repeat themselves much in their internal chain of thought, since that information is already available to them. The author of the analysis can’t read frontier chains of thought, but expects this from open models and general principles. Some early open-source reasoning models do loop, though they also use far more tokens than frontier models for the same answer.

So how much of the output is visible repetition? In the author’s own agent histories, visible output is about 6% of the total. Reasoning tokens make up 50%, and tool calls account for 44%.

That runs into Amdahl’s law: speeding up one small part of a system limits the total gain. Even an optimistic cut of two-thirds of visible output would reduce total output tokens by only about 4%.

One more detail: coding agents once re-output a whole file for each edit, but most have moved to find-and-replace editing tools. That removed one of the biggest sources of repeated tokens already.

The point of the exercise is to match the optimization to the cost. Output caching targets a segment that is easy to see and easy to complain about, but it isn’t where the tokens go. Most of the spending sits in work the user never reads.

Where the real savings are

The other 94% of output is invisible to the user, and cutting it mostly comes down to model intelligence. A smarter model needs fewer reasoning tokens and fewer tool calls to reach the right answer. Better surrounding tooling helps too, though the author doubts it gives a lasting edge because rivals can copy it.

That explains a counterintuitive pattern. A pricier model can be cheaper to use. If a model costs five times more per token but needs six times fewer tokens to get the answer, the total bill is lower.

What this means for agent costs

Teams trying to cut agent spending shouldn’t expect much from output caching. Reducing visible repetition would shave a few percent at best. Choosing a stronger model for hard tasks, or improving how often an agent gets the answer right, moves far more of the bill.

Some open questions remain. Tool calls might involve repetition if agents draft content in their reasoning and then copy it into a tool call. People who work with open-source models, or at frontier labs, would know whether that is common. Until there’s data showing it is, per-token price comparisons will keep misleading anyone who ignores how many tokens each model needs.

Source: seangoedecke.com

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.