Table of Contents
- The Assumption That Bigger Is Better
- Lost in the Middle
- Attention Dilution Is Real
- What Long Context Costs
- Positioning Matters More Than Volume
- When Long Context Genuinely Wins
- Prompt Caching Changes the Arithmetic
- A Practical Context Budget
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: A model that accepts a million tokens does not attend to a million tokens equally. Relevance density matters more than volume, and adding marginally relevant context measurably degrades output on the information that mattered.
The Assumption That Bigger Is Better
When context windows expanded from a few thousand tokens to hundreds of thousands and beyond, the natural response was to use the space. Include the whole document. Attach every related file. Paste the full conversation history. Why select when you can supply everything?
This intuition is wrong in a specific and measurable way, and the reason is that context capacity and context utility are different properties.
Capacity is what the model will accept without error. Utility is how reliably the model finds and uses a given piece of information within what it accepted. These diverge substantially, and they diverge most at exactly the scale where the extra capacity seemed most attractive.
The practical consequence is uncomfortable for anyone who has built a system around stuffing context: adding information that is only tangentially relevant can make the model worse at using the information that was highly relevant. You are not adding a neutral safety margin. You are introducing competition for attention.
Lost in the Middle
The most robustly documented effect in long-context behaviour is positional. Information at the start and end of a context window is recalled substantially more reliably than information in the middle.
This produces a U-shaped performance curve. Place a needed fact at the very beginning of a long context and retrieval accuracy is high. Place it at the very end and accuracy is high. Place it at the 50 percent mark and accuracy drops noticeably — in some published evaluations, enough that a model with a long context performs worse than the same model given a short, well-targeted context containing only the relevant passage.
Several architectural factors contribute. Positional encoding schemes generalise imperfectly to positions far beyond those seen frequently in training. Attention patterns learned during training show measurable bias toward sequence boundaries. And the middle of a long sequence simply competes against more surrounding tokens for attention weight than either edge does.
The immediate practical implication is that the arrangement of your context is a design decision with measurable consequences, not an incidental detail. If you have twenty documents and one contains the answer, where that document sits in your prompt affects whether the model uses it.
Attention Dilution Is Real
Beyond position, there is a volume effect that operates independently.
Attention is a normalised distribution — the weights across all tokens sum to a fixed total. Adding tokens necessarily reduces the average weight available per token. When the added tokens are relevant, this trade is fine. When they are marginal, you have spent attention budget on noise.
The observable symptoms of dilution are recognisable once you know to look for them. The model produces a plausible answer drawn from a nearby but incorrect passage. It hedges more, because contradictory information appeared somewhere in the context. It ignores a specific instruction, because that instruction was one line among fifty thousand tokens. It answers a subtly different question than the one asked.
The counterintuitive part is that these failures look like reasoning failures rather than retrieval failures. The natural response is to upgrade the model or elaborate the prompt. The effective response is usually to remove context.
A diagnostic worth running: take a case where output quality is poor, cut the context to only the passages a human would consider necessary, and compare. If quality improves substantially, the problem was dilution rather than capability — and no model upgrade would have fixed it.
What Long Context Costs
Three costs scale with context length, and only one of them is obvious.
Token cost scales linearly. Every request pays for every input token. A prompt that includes 200,000 tokens of context to answer a question that needed 2,000 costs a hundred times more than necessary, on every single call. At any meaningful volume this dominates the economics of the feature.
Latency scales worse than linearly. Time to first token grows with input length because the entire prompt must be processed before generation begins. For interactive features, a long prompt can push perceived response time from acceptable to unusable regardless of how fast generation itself is.
Attention computation scales quadratically in the general case. Modern implementations mitigate this considerably, and the underlying cost pressure remains — which is why long-context requests are both slower and more expensive per token than short ones.
| Context size | Relative token cost | Relative time to first token | Retrieval reliability |
|---|---|---|---|
| 2K tokens | 1× | 1× | Very high |
| 20K tokens | 10× | ~3× | High |
| 100K tokens | 50× | ~10× | Moderate, position-dependent |
| 500K tokens | 250× | ~40× | Variable, middle degrades |
The pattern to notice: cost rises strictly, latency rises steeply, and reliability does not improve past a point — it declines. That combination makes maximal context a poor default rather than a safe one.
Positioning Matters More Than Volume
Given the position effect, layout becomes an actionable lever. A structure that works reliably:
[1] System instructions and role definition
[2] Output format requirements and constraints
[3] Reference material, most relevant first
...
least relevant last
[4] Restated key constraints (brief)
[5] The actual user questionThe reasoning behind each choice:
Instructions first. They are needed for the entire generation and benefit from the strong recall at the beginning.
Most relevant material early. If reference material must be truncated or partially ignored, you want that to happen to the least useful content. Ordering by relevance makes degradation graceful.
Restate critical constraints near the end. A short repetition of the two or three things that absolutely must hold — output format, forbidden content, required fields — placed just before the question, exploits the strong end-position recall. This is cheap and measurably effective.
Question last. It should be the most recent thing the model attended to before generating.
Two further techniques help. Delimiting sections explicitly with tags or headers gives the model structural anchors and lets you refer to sections in your instructions. And labelling each document with a title or source makes citation possible and helps the model distinguish between similar passages — which is precisely where dilution failures occur.
When Long Context Genuinely Wins
None of this argues against long context. It argues against using it reflexively. There are cases where it is clearly the right tool.
Whole-document reasoning. Questions requiring synthesis across an entire document — summarise this contract, find inconsistencies between these sections, trace how this argument develops — cannot be answered from retrieved fragments. Retrieval returns passages; these tasks need the whole.
Codebase-wide changes. Understanding how a change ripples through a repository requires seeing the relationships. Chunked retrieval loses exactly the cross-file structure that matters.
Long conversation continuity. Multi-turn interactions where earlier context genuinely matters. Summarising history loses specifics that later turns may depend on.
Many-shot prompting. Supplying dozens or hundreds of examples improves performance on some tasks in ways a handful cannot. This is a legitimate and underused application of large windows.
Avoiding retrieval infrastructure entirely. For a corpus small enough to fit in context, skipping the embedding pipeline, vector store, and reranker is a real simplification. The cost is per-request tokens; the saving is an entire subsystem. For low-volume applications this trade frequently favours long context.
The distinguishing question is whether your task needs relationships across the whole input or specific facts from somewhere in it. The first favours long context. The second favours retrieval.
Prompt Caching Changes the Arithmetic
Prompt caching deserves separate treatment because it materially alters the cost calculation described above.
The mechanism: a stable prefix of your prompt is processed once and its internal representation cached. Subsequent requests reusing that exact prefix skip the recomputation, at substantially reduced cost and latency.
This makes a large, stable context economical in a way it otherwise is not. A 100,000-token document set that many queries run against becomes cheap after the first request, provided the prefix is byte-identical every time.
Designing for it requires discipline:
CACHEABLE PREFIX (identical on every request)
system instructions
stable reference documents
few-shot examples
──────────────────────────────
VARIABLE SUFFIX (changes per request)
conversation history
user questionThe critical constraint is that any change within the cached prefix invalidates it from that point onward. A timestamp, a session identifier, or a user name inserted near the top of an otherwise stable prompt destroys the caching benefit for everything after it. Variable content belongs at the end, without exception.
This changes advice meaningfully. Where caching is available and your context is stable across many requests, the cost argument against long context largely dissolves. The position and dilution effects do not — those are properties of the model’s attention, and no caching scheme addresses them.
A Practical Context Budget
A decision procedure that produces reasonable defaults:
Start with what a competent human would need. If a person answering this question would read three paragraphs, three paragraphs is your target. This is a better anchor than the model’s maximum.
Add context only with evidence. Each addition should be justified by a measured improvement on an evaluation set, not by the intuition that it might help.
Measure the reduction, not only the addition. Try removing context and observe what breaks. Teams routinely discover that a third of their prompt contributes nothing.
Use retrieval to select, then long context to reason. These are complementary rather than competing. Retrieve broadly, rerank hard, then give the model the surviving material with room to reason across it.
Reserve capacity for output. The context window is shared between input and generation. Filling 95 percent with input leaves insufficient room for a long response and can cause truncation.
Common Pitfalls
Filling the window because it exists. Capacity is a limit, not a target.
Variable content in the cacheable prefix. A single injected timestamp near the top eliminates caching for the entire remainder.
Burying critical instructions mid-prompt. They will be followed inconsistently. Move them to the start or repeat them at the end.
Assuming retrieval became unnecessary. For fact-finding over large corpora, targeted retrieval remains both cheaper and more accurate than long-context search.
No evaluation of context changes. Adding context is a change to system behaviour and deserves the same regression testing as a code change.
Conclusion
Expanded context windows are a genuine capability increase and a poor default. The model that accepts a million tokens does not weight them equally: position affects recall, volume dilutes attention, and both cost and latency scale against you.
Treat context as a budget rather than a container. Order material by relevance so that degradation is graceful. Put instructions at the beginning, restate the critical ones at the end, and place the question last. Structure prompts so the stable portion can be cached and the variable portion sits at the end.
Then measure. The most common finding when teams first evaluate context length seriously is that a shorter, better-organised prompt outperforms the long one it replaced — at a fraction of the cost.
Frequently Asked Questions
Has long context made RAG obsolete? No. For corpora that fit entirely in context and low request volume, long context is simpler. For large corpora, high volume, or applications needing citations to specific sources, retrieval remains both cheaper and more accurate.
Where should the most important information go? Beginning or end. The middle of a long context is where recall is weakest. If one fact matters more than everything else, place it at the start and restate it briefly before the question.
Does prompt caching remove the cost concern? Largely, for stable prefixes reused across many requests. It does not address position effects or attention dilution, which are model behaviours rather than pricing artefacts.
How do I know if my context is too long? Remove half and measure on an evaluation set. If quality holds or improves, it was too long. This experiment is cheap and frequently surprising.
Should conversation history be summarised or kept verbatim? Verbatim for recent turns, where specifics matter. Summarised for older turns, where the gist suffices. A rolling window of full recent history plus a running summary of what preceded it works well in practice.
Do all models show the same position effects? The direction is consistent across models — edges recalled better than middle — while the magnitude varies considerably. Testing on the specific model you deploy is worthwhile rather than assuming published results transfer.
Is there a downside to short context? Yes: omitting information the model needed. The failure is different and easier to diagnose, since the model states it lacks information rather than confidently using the wrong passage. That diagnosability is itself an argument for erring shorter.
