What is prompt caching and how does it work?
Also called prefix caching, context caching, prompt cache
Definition
Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
Prompt caching is a technique in which an LLM provider or inference server stores the intermediate attention state for the beginning of a prompt and reuses it when a later request starts with exactly the same tokens. The stored data is the KV cache computed during prefill, not the model's answer, so every response is still generated fresh. Providers use different names for the same idea: Anthropic and OpenAI call it prompt caching, vLLM calls it automatic prefix caching, and Google calls it context caching.
How prompt caching works#
A cache hit requires an identical prefix. The server matches the incoming prompt against stored prefixes from the first token forward, loads the keys and values for the longest match, and computes attention only for the tokens after that point.
The controls differ by provider. On the Anthropic API, developers mark breakpoints with cache_control; the default cache lifetime is five minutes, refreshed each time the entry is read, and a one-hour lifetime is available. OpenAI caches automatically on supported models once a prompt passes a minimum length, and reports hits in the cached_tokens usage field. vLLM hashes each full block of KV cache together with every token before it and evicts unused blocks least recently used first.
The layout rule is the same everywhere: put stable content (instructions, tool definitions, reference documents) first and variable content (timestamps, user input) last. One changed token early in the prompt invalidates the cache for everything after it.
Why prompt caching matters#
Prompt caching lowers time to first token because cached tokens skip prefill, and prefill time grows with prompt length. Prompt caching also lowers input cost on providers that bill cached reads at a discount. Agents, chat applications with long system prompts, and retrieval pipelines that resend the same documents benefit most, since they repeat large identical prefixes on every turn. The cache's prefix rule is why context engineering treats prompt ordering as a performance decision as well as a quality one, and why inference optimization work often starts with cache hit rates.
Prompt caching vs semantic caching#
Semantic caching stores complete responses and returns a stored answer when a new query is similar in meaning, so the model is not called at all. Prompt caching always calls the model and reuses only the input-side computation. A semantic cache can return an answer that does not fit the new query. Prompt caching does not change the output, because a hit requires an exact token match.
Related terms
Guides that cover prompt caching
- LLM inference optimization: a practical guide for engineers
- What is context engineering? A practical guide for AI agents
- Semantic caching for LLMs: how it works and when it pays off
- How to build an AI agent: a step-by-step guide in Python
- LLM observability: what to capture, which metrics, and tools
- LLM routing: strategies, fallbacks, and learned routers
- What is an AI gateway? How it works and when you need one
- Reasoning models: how they work and when to use them
- Chunking strategies for RAG: how to choose, tune, and evaluate them
- RAG vs fine-tuning: how to choose for your LLM application
- What is retrieval-augmented generation (RAG)? How RAG works
Cite this definition
The AI Engineering Editors. "What is prompt caching and how does it work?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prompt-caching
- Title
- What is prompt caching and how does it work?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated