---
title: "What is context rot in LLMs?"
description: "Context rot is the drop in LLM accuracy as input grows, even when the answer is in the context. The research behind it and how it differs from window limits."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/context-rot"
markdown_url: "https://theaiengineering.com/glossary/context-rot.md"
topic: "Prompt and context engineering"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Context rot","lost in the middle","long-context degradation"]
citation: "The AI Engineering Editors. \"What is context rot in LLMs?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/context-rot"
---

# What is context rot in LLMs?

Also called lost in the middle, long-context degradation.

> **Definition of Context rot:** Context rot is the decline in a language model's accuracy and recall as the number of tokens in its input grows, even when the information a task needs is present in the context.

Context rot is the decline in a model's ability to find and use information accurately as its input grows longer. The information can be present and well inside the [context window](https://theaiengineering.com/glossary/context-window), yet the model misses it or misuses it more often as more tokens surround it. Chroma's July 2025 technical report, titled [Context Rot](https://www.trychroma.com/research/context-rot), measured the effect across 18 models, and Anthropic's engineering guidance uses the same term. For an AI engineer, context rot means that choosing what goes into a prompt affects answer quality as well as cost.

## Evidence for context rot

Two studies anchor the term. Liu et al.'s 2023 paper Lost in the Middle found that accuracy depends on where relevant information sits in a long input. Chroma's 2025 report found that accuracy falls as input grows, even on simple retrieval and copying tasks, across models from Anthropic, OpenAI, Google, and Alibaba.

[Lost in the Middle](https://arxiv.org/abs/2307.03172) (Liu et al., published in TACL) tested multi-document question answering and key-value retrieval. The authors report that performance is often highest when the relevant information is at the beginning or end of the input and degrades significantly when it sits in the middle, including for models built for long contexts. The position effect is widely called the lost-in-the-middle problem.

The Chroma report, by Kelly Hong, Anton Troynikov, and Jeff Huber, added findings that matter for agents:

- Accuracy fell faster as input grew when the question and the relevant passage were less semantically similar.
- A single distractor, a passage on the same topic that does not contain the answer, lowered accuracy compared with the relevant passage alone.
- Haystacks with shuffled sentences produced better results than logically coherent ones, so the structure of the surrounding text affects retrieval.
- On the LongMemEval conversational benchmark, every model scored significantly higher on focused prompts of about 300 tokens than on full prompts of about 113,000 tokens.

Anthropic's [context engineering post](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) connects the effect to attention. Self-attention relates every token to every other token, which gives n² pairwise relationships for n tokens, and the post calls context a finite resource with diminishing marginal returns.

## Context rot vs context window limits

A context window limit is a hard cap: input beyond it is rejected or must be cut. Context rot is a gradual decline that can appear long before the cap, as more tokens reduce how reliably the model uses the relevant ones. A request can fit comfortably inside the window and still lose accuracy to context rot.

| | Context window limit | Context rot |
|---|---|---|
| Kind | Hard limit set by the model | Gradual loss of accuracy |
| Symptom | An error or a truncated input | Missed or misused facts that are in the context |
| Made worse by | Exceeding the token limit | More tokens, distractors, key facts in the middle |
| Remedy | Fit the input under the limit | Send fewer, more relevant tokens, with key facts at the start or end |

A larger window raises the cap without removing the decline, so long-context models still benefit from careful selection. Teams reduce context rot with [context compaction](https://theaiengineering.com/glossary/context-compaction) for long histories, with [retrieval-augmented generation](https://theaiengineering.com/glossary/retrieval-augmented-generation) and [reranking](https://theaiengineering.com/glossary/reranking) to send only the strongest passages, and by clearing stale tool results. Placing critical information at the start or end of the input follows from the Lost in the Middle results.

The [context engineering guide](https://theaiengineering.com/articles/context-engineering) covers these techniques and how to measure their effect.

## Related terms

- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Context engineering](https://theaiengineering.com/glossary/context-engineering): Context engineering is the practice of selecting, structuring, and pruning all the information placed in an LLM's context window at each step, including instructions, retrieved data, tool results, and memory.
- [Context compaction](https://theaiengineering.com/glossary/context-compaction): Context compaction is a context management technique that replaces older conversation turns with a model-written summary as the context window nears its limit, so an agent can keep working.
- [Retrieval-augmented generation (RAG)](https://theaiengineering.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique that retrieves relevant passages from an external knowledge source at query time and adds them to a language model's prompt so the answer is grounded in that data.
- [Reranking](https://theaiengineering.com/glossary/reranking): Reranking is a second retrieval stage that rescores an initial list of candidate documents with a more accurate but slower model, usually a cross-encoder, and keeps only the most relevant ones.
- [Agent memory](https://theaiengineering.com/glossary/agent-memory): Agent memory is the set of mechanisms that let an AI agent keep and retrieve information beyond a single model call, from the current context window to external stores that persist across sessions.

## Guides that cover context rot

- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [What is retrieval-augmented generation (RAG)? How RAG works](https://theaiengineering.com/articles/what-is-rag): Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes.

Part of the [Prompt and context engineering](https://theaiengineering.com/topics/prompt-engineering) topic hub.

## Cite this definition

The AI Engineering Editors. "What is context rot in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/context-rot

- Title: What is context rot in LLMs?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/context-rot
- Markdown: https://theaiengineering.com/glossary/context-rot.md
