What is context rot in LLMs?
Also called lost in the middle, long-context degradation
Definition
Context rot is the decline in a language model's accuracy and recall as the number of tokens in its input grows, even when the information a task needs is present in the context.
Context rot is the decline in a model's ability to find and use information accurately as its input grows longer. The information can be present and well inside the context window, yet the model misses it or misuses it more often as more tokens surround it. Chroma's July 2025 technical report, titled Context Rot, measured the effect across 18 models, and Anthropic's engineering guidance uses the same term. For an AI engineer, context rot means that choosing what goes into a prompt affects answer quality as well as cost.
Evidence for context rot#
Two studies anchor the term. Liu et al.'s 2023 paper Lost in the Middle found that accuracy depends on where relevant information sits in a long input. Chroma's 2025 report found that accuracy falls as input grows, even on simple retrieval and copying tasks, across models from Anthropic, OpenAI, Google, and Alibaba.
Lost in the Middle (Liu et al., published in TACL) tested multi-document question answering and key-value retrieval. The authors report that performance is often highest when the relevant information is at the beginning or end of the input and degrades significantly when it sits in the middle, including for models built for long contexts. The position effect is widely called the lost-in-the-middle problem.
The Chroma report, by Kelly Hong, Anton Troynikov, and Jeff Huber, added findings that matter for agents:
- Accuracy fell faster as input grew when the question and the relevant passage were less semantically similar.
- A single distractor, a passage on the same topic that does not contain the answer, lowered accuracy compared with the relevant passage alone.
- Haystacks with shuffled sentences produced better results than logically coherent ones, so the structure of the surrounding text affects retrieval.
- On the LongMemEval conversational benchmark, every model scored significantly higher on focused prompts of about 300 tokens than on full prompts of about 113,000 tokens.
Anthropic's context engineering post connects the effect to attention. Self-attention relates every token to every other token, which gives n² pairwise relationships for n tokens, and the post calls context a finite resource with diminishing marginal returns.
Context rot vs context window limits#
A context window limit is a hard cap: input beyond it is rejected or must be cut. Context rot is a gradual decline that can appear long before the cap, as more tokens reduce how reliably the model uses the relevant ones. A request can fit comfortably inside the window and still lose accuracy to context rot.
| Context window limit | Context rot | |
|---|---|---|
| Kind | Hard limit set by the model | Gradual loss of accuracy |
| Symptom | An error or a truncated input | Missed or misused facts that are in the context |
| Made worse by | Exceeding the token limit | More tokens, distractors, key facts in the middle |
| Remedy | Fit the input under the limit | Send fewer, more relevant tokens, with key facts at the start or end |
A larger window raises the cap without removing the decline, so long-context models still benefit from careful selection. Teams reduce context rot with context compaction for long histories, with retrieval-augmented generation and reranking to send only the strongest passages, and by clearing stale tool results. Placing critical information at the start or end of the input follows from the Lost in the Middle results.
The context engineering guide covers these techniques and how to measure their effect.
Related terms
Guides that cover context rot
Cite this definition
The AI Engineering Editors. "What is context rot in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/context-rot
- Title
- What is context rot in LLMs?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated