What is a context window in LLMs?

Also called context length, context size, context limit

Definition

A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.

A context window is the maximum number of tokens a large language model can process in one request. Everything the model sees counts against it: the system prompt, messages, tool definitions, tool results, attached documents, and the response the model generates. Anthropic's documentation describes the context window as the model's working memory, distinct from the data it was trained on.

How a context window works#

A context window works as a hard token budget per request. The application assembles a prompt, the model processes it, and each generated token is added to the same budget until the response ends or the limit is reached. In a multi-turn chat, every turn resends the earlier history, so the window fills as the conversation grows.

When the input alone exceeds the window, the API rejects the request; Anthropic's API, for example, returns a prompt-too-long error. Applications handle long inputs by dropping old turns, summarizing history (often called compaction), or retrieving only relevant passages with retrieval-augmented generation. Prompt caching makes a repeated prefix cheaper to process, but cached tokens still occupy the window.

Context window vs max tokens#

The context window and max tokens are two different limits. The context window is the model's total capacity for input plus output, fixed by the model. Max tokens, set with a request parameter such as max_tokens, is a cap on output length chosen by the caller, and the output must fit in whatever room the input leaves.

Context window Max tokens
Set by The model The API caller
Covers Input and output Output only
When exceeded Request rejected, or generation stopped at the limit Output stops early

Why the context window matters#

The context window matters because it bounds how much information a model can use at once, yet filling it has costs. Lost in the Middle (Liu et al., 2023) found that models use information at the start and end of a long input more reliably than information in the middle. Anthropic's context window documentation notes that accuracy and recall degrade as token count grows, a pattern it calls context rot. Long inputs also raise cost and latency, because attention compute and the KV cache grow with sequence length.

A larger window is therefore a ceiling, not a target. Context engineering covers how to choose what goes into the window and what stays out.

Cite this definition

The AI Engineering Editors. "What is a context window in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/context-window

Title
What is a context window in LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI engineering hub · Updated