Do long context agents really need million-token windows? A new Nvidia research preprint says mostly no. Across 13 benchmarks, agents capped at 48,000 tokens of context performed about as well as agents carrying their whole history, and on big documents they frequently came out ahead. The work hasn’t been peer reviewed.
Window sizes have stopped growing
OpenAI’s flagship shipped in March 2023 with around 8,000 tokens. By early 2024 Google was previewing a million. Today’s top models from OpenAI, Anthropic and Google still sit near that figure, while a few rivals advertise two or ten million.
The size plateaued. The belief that more visible context makes a better agent didn’t, and Nvidia’s team went after it.
Discarding history barely hurts
Researchers ran one simple agent in three modes. The baseline keeps every command and tool output until the task ends. The second mode lets context fill to a set limit, then has the agent write down its working state and restart fresh with only the task and that note. The third takes one action, updates the note and wipes the conversation. Files on disk stay; the transcript goes.
Six suites covered long multi-step work: real GitHub issues, terminal tasks, multi-hop research and tool calling under business rules. Two frontier closed-weight models landed between 3.7 points below and 5.6 points above their baselines. Across 22 benchmark and budget combinations per model, each lost more than two points in at most five.
The one-action mode was the most striking. On 500 real GitHub issues, one model scored 70.0 percent with full history and 70.2 with a wipe after every action. The other moved from 84.4 to 82.4. Peak context dropped from 130,000-200,000 tokens to 15,000-37,000.
The saved note matters. Plain summaries did worse, the authors say, since they treat every fact equally and overweight whatever the transcript repeated. Good notes read like a hand-off brief: the task word for word, where the agent stands, the plan and the artefacts still needed.
Search beats stuffing the prompt
Seven benchmarks use huge inputs. Instead of pasting documents into the prompt, the team left them as files and let the agent search and read excerpts of up to 8,000 characters.
On LOONG, with 1,600 multi-document questions, one model went from 70.1 on full text to 79.8 using search under a 48,000-token cap. On AA-LCR, 100 cross-document reasoning questions, the same model rose from 65 to 84 percent at 16,000 tokens. On RULER at 256,000 and 500,000 input tokens, recall climbed from the low sixties to above 95 for both models.
Gains weren’t universal. The second model improved by a single point on AA-LCR, and the first lost 0.4 on LongBench v2. Search was never clearly worse, and it won big where full text struggled.
Claude Code took this route earlier
Early Claude Code relied on embeddings and a local vector database. The team switched to agentic search, where the model runs tools like glob and grep and searches again as needed. Creator Boris Cherny said in a May 2025 interview it worked better and spared the team from keeping an index in sync. He called the evidence mostly vibes plus internal benchmarks, and admitted the price: extra latency and tokens.
The comparisons differ. Claude Code weighed a prebuilt index against model-steered search, while the Nvidia paper asks how much must sit in context at once. They agree that sources can stay outside the model if the agent can fetch them.
Cost claims rest on a model
Nvidia makes the hardware, so it gains if short contexts suffice. Its team served three open-weight models on H100 clusters and cut the window from 256,000 to 16,000 tokens, which lifted output throughput 3.6, 5.6 and 7 times. That shows a provider can serve more tokens per GPU. It doesn’t mean your bill shrinks by the same factor.
The headline savings, up to 233 times on a needle-in-a-haystack test, come from the authors’ own price model, which discounts closed models’ list prices by the measured throughput gains. No provider charges that way. At standard rates, one model’s LongBench v2 bill rose from $79.20 to $90.20 when it searched.
Where it falls short
- Every model was built with a large window and then restricted, so models trained only on short contexts remain untested.
- One open-weight coding model collapsed from 53.0 to 3.6 percent on the GitHub suite at the tightest budget, largely because verbose transcripts crowded out room.
- In a separate test on 102 contracts, whole documents beat classic RAG by 3.4 to 4.6 F1 points, and retrieval saved only about 11 percent.
Compaction loses detail by design. The authors’ example: an agent logging coin tosses will eventually drop one someone asks about later.
What builders should try
For anyone building long context agents, the advice is practical. Keep sources on disk with a size-limited reader, inside a sandbox. Write each compaction as a brief for the next conversation. Store the full transcript outside the context and make it searchable. Test your model under the budget first.
One caveat remains: every benchmark question tells the agent what to look for, and real work often doesn’t.
Source: IKANGAI