What is prompt caching and how does it work?

Also called prefix caching, context caching, prompt cache

Definition

Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.

Prompt caching is a technique in which an LLM provider or inference server stores the intermediate attention state for the beginning of a prompt and reuses it when a later request starts with exactly the same tokens. The stored data is the KV cache computed during prefill, not the model's answer, so every response is still generated fresh. Providers use different names for the same idea: Anthropic and OpenAI call it prompt caching, vLLM calls it automatic prefix caching, and Google calls it context caching.

How prompt caching works#

A cache hit requires an identical prefix. The server matches the incoming prompt against stored prefixes from the first token forward, loads the keys and values for the longest match, and computes attention only for the tokens after that point.

Two requests share an identical prefix of system prompt, tool definitions, and document; request 2 reads that prefix from the KV cache and computes only question B. Request 1Request 2Shared prefixSystem prompt · tool defs · documentComputed, then cachedQuestion AComputedShared prefixSystem prompt · tool defs · documentIdentical tokens, read from cacheQuestion BComputedKV cache reused
Figure 1: Only the tokens after the shared prefix are processed again.

The controls differ by provider. On the Anthropic API, developers mark breakpoints with cache_control; the default cache lifetime is five minutes, refreshed each time the entry is read, and a one-hour lifetime is available. OpenAI caches automatically on supported models once a prompt passes a minimum length, and reports hits in the cached_tokens usage field. vLLM hashes each full block of KV cache together with every token before it and evicts unused blocks least recently used first.

The layout rule is the same everywhere: put stable content (instructions, tool definitions, reference documents) first and variable content (timestamps, user input) last. One changed token early in the prompt invalidates the cache for everything after it.

Why prompt caching matters#

Prompt caching lowers time to first token because cached tokens skip prefill, and prefill time grows with prompt length. Prompt caching also lowers input cost on providers that bill cached reads at a discount. Agents, chat applications with long system prompts, and retrieval pipelines that resend the same documents benefit most, since they repeat large identical prefixes on every turn. The cache's prefix rule is why context engineering treats prompt ordering as a performance decision as well as a quality one, and why inference optimization work often starts with cache hit rates.

Prompt caching vs semantic caching#

Semantic caching stores complete responses and returns a stored answer when a new query is similar in meaning, so the model is not called at all. Prompt caching always calls the model and reuses only the input-side computation. A semantic cache can return an answer that does not fit the new query. Prompt caching does not change the output, because a hit requires an exact token match.

Cite this definition

The AI Engineering Editors. "What is prompt caching and how does it work?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prompt-caching

Title
What is prompt caching and how does it work?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated