What is context engineering? A practical guide for AI agents

Explainer · Context

Context engineering is the practice of deciding which information fills a language model's context window at each step of a task: the instructions, tool definitions, retrieved documents, tool results, memory, and conversation history. Context engineering matters most for AI agents, because an agent rebuilds that context on every model call during a run and every token adds cost, latency, and noise.

This explainer defines the term, compares it with prompt engineering, breaks the context window into its components, reviews the research on long-context degradation, and covers the techniques and metrics teams use in production.

What is context engineering?#

Context engineering is the design of systems that select, format, and maintain the tokens a language model receives at inference time, so each call has what the task needs. Anthropic's engineering team frames the goal as finding the smallest set of high-signal tokens that produces the desired outcome, and treats context as a finite resource.

The term spread in June 2025, when Shopify CEO Tobi Lütke and former OpenAI researcher Andrej Karpathy both posted that they preferred "context engineering" to "prompt engineering." Karpathy's argument was that people associate prompts with short task descriptions, while production LLM applications must fill the context window with much more than that. Anthropic's September 2025 guide, Effective context engineering for AI agents, then described the practice in detail for agent builders.

Context engineering vs prompt engineering#

Prompt engineering is the craft of writing instructions for a model; context engineering is the broader job of managing everything in the context window, including those instructions. A prompt is usually written once and tested. An agent's context is assembled again on every model call from changing sources, so context engineering resembles systems design more than writing.

Dimension Prompt engineering Context engineering
Main object Instructions and examples Every token in the window
When it happens Design time Every model call at runtime
Typical owner Whoever writes the prompt Engineers building retrieval, memory, and tool layers
Common failure Ambiguous or conflicting instructions Missing facts, stale or distracting content, overflow
Main methods Wording, structure, few-shot examples Retrieval, compaction, memory stores, sub-agents, caching
How to test Prompt variants on an eval set Ablations of context sources, plus token and cache metrics

Prompt engineering does not go away. The system prompt and tool descriptions are still written by hand, and their wording still changes results. Context engineering adds the runtime decisions around them: which documents to retrieve, which tool output to keep, and what to summarize or drop as the window fills.

Why context engineering is important for AI agents#

Context engineering became important with AI agents because agents accumulate context as they work. Each tool call adds its arguments and results to the conversation, so a long task can fill the window with stale file contents, logs, and search results. That growth raises cost and latency, and it lowers accuracy unless something removes or condenses the excess.

A chat application sends one question and receives one answer. An agent runs a loop of model calls, and each call re-sends the accumulated context as input tokens. The Manus team, writing about its production agent in July 2025, reported an average input-to-output token ratio of around 100 to 1. In that profile, input tokens account for most of the processing, which is why the reuse of a cached prefix through prompt caching and the KV cache matters so much for agents.

Quality is the second reason. Anthropic's guide describes an "attention budget": in a transformer, every token attends to every other token, which creates n² pairwise relationships for n tokens, and each added token draws down that budget. The AI agents explainer covers the loop itself; context engineering is the discipline of keeping that loop's input small and relevant.

What goes into the context window?#

An agent's context window holds six kinds of content: the system prompt, tool definitions, memory loaded for the task, retrieved context, conversation history, and tool calls with their results. Each has a different source and growth pattern, so each needs its own policy for what to include, when to include it, and in what form.

Seven stacked layers of an agent context window, from the stable system prompt and tool definitions at the top down to fast-growing tool results and the latest message. Most stableSystem prompt and project instructionsStableTool definitionsStable · names and JSON schemasMemory loaded for this taskPer taskRetrieved documents and file contentsPer stepConversation history or compacted summaryGrowsTool calls and tool resultsGrows fastLatest user message or next instructionPer stepMost volatile
Figure 1: A typical agent context window, ordered from most stable to most volatile, which keeps the cached prefix valid.
Component Source Growth pattern Main control
System prompt Written by developers Fixed Specific rules, no contradictions
Tool definitions Application code or MCP servers Grows with each integration Fewer, distinct tools; defer rare ones
Memory Notes files, databases, memory tools Loaded selectively Write summaries; load by relevance
Retrieved context Search, RAG, file reads Per step Retrieve just in time; rerank; cap size
Conversation history User and model turns Steady Compact near the limit
Tool results Tool execution Fast, every step Truncate, paginate, clear old results

Tool definitions#

Tool definitions are easy to overlook because they do not appear in the conversation. OpenAI's function calling guide notes that function definitions are injected into the system message, count against the context limit, and are billed as input tokens. The same guide suggests keeping fewer than 20 functions available at the start of a turn as a soft target, and deferring rarely used tools with tool search. Each MCP server an agent connects to adds more definitions, so the tool calling guide treats tool count as a design decision.

A gateway can trim the list before it reaches the model. Bifrost filters MCP tools per virtual key, and its Code Mode replaces the full tool catalog with four meta-tools that the model uses to look up tool signatures and run a short Python-like script against them. In Bifrost's own benchmark with 508 tools across 16 servers, Code Mode cut input tokens by 92.8% at the same pass rate.

Memory#

Agent memory is information stored outside the window and loaded back when relevant. Short-term memory is the working state of the current task. Long-term memory persists across sessions in notes files, databases, or provider features such as Anthropic's memory tool. Coding agents use project files for the same purpose: Claude Code reads CLAUDE.md at the start of every session and can save its own notes across sessions.

Conversation history#

Conversation history is the record of user turns, model turns, and tool exchanges. History is the component that eventually overflows the window, so most agents need a compaction policy that decides when to summarize and what the summary must preserve.

What is context rot?#

Context rot is the decline in a model's accuracy as the amount of input grows, even when the information the task needs is present. Chroma used the term as the title of a July 2025 research report, and Anthropic's context window documentation now uses it to explain why curating context matters as much as the size of the window.

Three lines of research support the finding:

Study What it tested Key finding
Lost in the Middle (Liu et al., 2023) Multi-document QA and key-value retrieval Accuracy is highest when relevant information is at the start or end of the input and drops when it sits in the middle
NoLiMa (Modarressi et al., 2025) Retrieval with minimal word overlap between question and answer At 32K tokens, 11 of 13 models fell below 50% of their short-context scores
Context Rot (Hong, Troynikov, and Huber, Chroma, 2025) 18 models on needle retrieval, conversational QA, and word replication Performance changes with input length even on simple tasks, and a single distractor lowers accuracy

The Chroma report adds two details that matter for agents. Accuracy falls faster as input grows when the question and the relevant passage are less similar, a common situation in agent work. Distractors, meaning passages that are topically related to the answer but do not contain it, lower accuracy, and more distractors lower it further; agent contexts are full of such near-miss content, including earlier versions of the same file and logs from previous test runs.

The practical conclusion is that a context window size is a capacity limit, not a quality guarantee. Models that accept a million tokens still benefit from receiving only the tokens a step needs. The Lost in the Middle results also argue for placing critical information at the start or end of the input, which the recitation technique described below applies to an agent's goals.

How to manage the context window: core techniques#

Context engineering techniques fall into five groups: select only what the current step needs, compress what has to stay, persist information outside the window, isolate subtasks in separate contexts, and order the window so caches stay valid. The groups are complementary, because each one addresses a different failure.

Select: retrieve just in time#

Instead of loading a codebase or document set up front, the agent keeps lightweight references such as file paths, search queries, or URLs and fetches content with tools when a step needs it. Retrieval-augmented generation applies the same idea to knowledge bases, and reranking keeps only the strongest passages. Tools should return the smallest useful result: paginated lists, filtered fields, and truncated logs with a pointer to the full output.

Compress: compaction and tool result clearing#

Compaction summarizes the conversation as it nears the window limit and continues from the summary. Tool result clearing is a lighter form that removes old raw tool outputs while keeping the record that the call happened. Both are now provider features: Anthropic's context editing clears the oldest tool results on the server once a threshold is passed, and OpenAI's Responses API offers server-side compaction and a standalone compaction endpoint. Compaction is lossy, so keep it restorable: drop a page's content but keep its URL, and drop a file's content but keep its path.

Persist: write notes outside the window#

Structured note-taking lets an agent record decisions, progress, and open questions in a file or memory store, then reload them after compaction or in a new session. The Manus team describes having its agent rewrite a todo.md file as it works, which keeps the current objectives near the end of the context.

Isolate: use sub-agents with clean context#

A sub-agent receives a narrow task in a fresh context window, does the exploration, and returns a condensed result to the lead agent. The lead agent's context then holds a short answer instead of every file the sub-agent read. The cost is coordination overhead and the risk that the summary drops a detail the lead agent needed.

Order: keep the prefix stable#

Prompt caches and KV caches reuse computation only for an identical prefix. Keep the system prompt and tool definitions first and unchanged, append new content instead of editing earlier turns, serialize data deterministically, and avoid placing changing values at the top; the Manus team names a timestamp at the start of the system prompt as a common mistake. The Manus team also recommends masking unavailable tools rather than removing their definitions mid-task, and leaving failed actions and error messages in the context so the model's next step accounts for them.

Context engineering example: a coding agent fixing a bug#

A coding agent fixing a failing test shows the techniques working together. The agent starts from a short, stable prefix, searches for what it needs instead of loading the repository, trims and clears test output, writes progress notes before compaction, and delegates broad searches to a sub-agent. The sequence below follows one run.

  1. Stable prefix. The run starts with the system prompt, the project's AGENTS.md file, and tool definitions, in the same order as every other run.
  2. Just-in-time retrieval. The agent runs the failing test, searches for the failing function name, and reads the two files that define and call that function.
  3. Trimmed results. The test tool returns the last 50 lines of output and writes the full log to a file the agent can open if needed.
  4. Clearing. After the second test run, earlier test output is replaced by a one-line record of the result.
  5. Notes. The agent updates a progress file with the hypothesis it tested and what failed.
  6. Isolation. A sub-agent searches for every caller of the changed function and returns a list of file paths.
  7. Compaction. Near the window limit, the history is summarized; the progress file and file paths are reloaded so the plan survives.

The AI coding agents comparison covers the tools that run loops like this one, and where each one executes.

How to measure context engineering#

Measure context engineering the same way as any other model change: run a fixed evaluation set before and after, and compare task success alongside cost and latency. Many context changes trade quality against tokens, so a change that halves input tokens but lowers success on long tasks is a regression, not an optimization.

Useful metrics include:

  • Task success rate, split by task length, because context problems show up on long runs first.
  • Input tokens per task and per step, including tool definitions and re-sent history.
  • Cache hit rate, which the Manus team calls the most important metric for a production agent because it drives both latency and cost.
  • Compaction outcomes: how often compaction runs and how often a task fails afterward because the summary dropped something.
  • Retrieval recall: whether the retrieved context contained the fact the step needed.
  • Ablations: remove one context source at a time and measure the change in success.

Traces make these metrics possible. The LLM observability guide covers how to capture each model call with its full input, and the LLM evaluation guide covers building the fixed task set.

Where to go next#

Context engineering connects to two adjacent skills: tool calling, which determines how much each tool adds to the window, and retrieval-augmented generation, which determines what external knowledge enters it. For more on prompts, context, and tool use, start from the prompt engineering topic hub.

Frequently asked questions

What is the difference between context engineering and prompt engineering?

Prompt engineering is the practice of writing clear instructions and examples for a model. Context engineering is the broader practice of deciding everything the model receives on each call, including those instructions, tool definitions, retrieved documents, tool results, memory, and history. Prompt engineering is usually done at design time, while context engineering runs at every step of an agent loop.

Is context engineering the same as RAG?

No. Retrieval-augmented generation is one context engineering technique: it selects external documents and inserts them into the prompt. Context engineering also covers tool definitions, tool results, memory, conversation history, compaction, and ordering. An agent can use RAG for knowledge lookups and still need separate policies for trimming logs, summarizing history, and isolating subtasks.

Does more context use more tokens?

Yes. Every token in the context window is processed on each model call and billed as input, including tool definitions and earlier turns that are re-sent. Prompt caching lowers the cost of a repeated prefix, but it does not reduce the window size. Larger inputs also add latency and, according to long-context research, often reduce accuracy.

What is agent memory?

Agent memory is information an AI agent stores outside its context window and loads back when needed. Short-term memory is the current conversation and working state. Long-term memory includes notes files, databases, or provider memory tools that persist across sessions. Context engineering decides what gets written to memory, when it is retrieved, and how much of it enters the window.

Is long context better than RAG?

Neither is better in general. Long context avoids retrieval errors when the full source fits and the task needs broad reading, but costs more per call and suffers from context rot. Retrieval keeps inputs short and focused but can miss relevant passages. Many production systems retrieve first, then use a long window for the retrieved set and the working history.

Terms in this guide

Cite this guide

The AI Engineering Editors. "What is context engineering? A practical guide for AI agents." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/context-engineering

Title
What is context engineering? A practical guide for AI agents
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·