---
title: "What is prompt caching and how does it work?"
description: "Prompt caching reuses the processed prefix of a repeated LLM prompt across requests to cut time to first token and input cost. How it works and what breaks it."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/prompt-caching"
markdown_url: "https://theaiengineering.com/glossary/prompt-caching.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Prompt caching","prefix caching","context caching","prompt cache"]
citation: "The AI Engineering Editors. \"What is prompt caching and how does it work?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prompt-caching"
---

# What is prompt caching and how does it work?

Also called prefix caching, context caching, prompt cache.

> **Definition of Prompt caching:** Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.

Prompt caching is a technique in which an LLM provider or inference server stores the intermediate attention state for the beginning of a prompt and reuses it when a later request starts with exactly the same tokens. The stored data is the [KV cache](https://theaiengineering.com/glossary/kv-cache) computed during [prefill](https://theaiengineering.com/glossary/prefill-and-decode), not the model's answer, so every response is still generated fresh. Providers use different names for the same idea: Anthropic and OpenAI call it prompt caching, vLLM calls it automatic prefix caching, and Google calls it context caching.

## How prompt caching works

A cache hit requires an identical prefix. The server matches the incoming prompt against stored prefixes from the first token forward, loads the keys and values for the longest match, and computes attention only for the tokens after that point.

**Figure 1: Only the tokens after the shared prefix are processed again.**

Two requests share an identical prefix of system prompt, tool definitions, and document; request 2 reads that prefix from the KV cache and computes only question B.

Components:

- **Shared prefix**: System prompt · tool defs · document Computed, then cached
- **Question A**: Computed
- **Shared prefix**: System prompt · tool defs · document Identical tokens, read from cache
- **Question B**: Computed

Groups:

- Request 1: Shared prefix, Question A
- Request 2: Shared prefix, Question B

Connections:

- Shared prefix → Shared prefix: KV cache reused

The controls differ by provider. On the [Anthropic API](https://platform.claude.com/docs/en/build-with-claude/prompt-caching), developers mark breakpoints with `cache_control`; the default cache lifetime is five minutes, refreshed each time the entry is read, and a one-hour lifetime is available. [OpenAI](https://developers.openai.com/api/docs/guides/prompt-caching) caches automatically on supported models once a prompt passes a minimum length, and reports hits in the `cached_tokens` usage field. vLLM hashes each full block of KV cache together with every token before it and evicts unused blocks least recently used first.

The layout rule is the same everywhere: put stable content (instructions, tool definitions, reference documents) first and variable content (timestamps, user input) last. One changed token early in the prompt invalidates the cache for everything after it.

## Why prompt caching matters

Prompt caching lowers [time to first token](https://theaiengineering.com/glossary/time-to-first-token) because cached tokens skip prefill, and prefill time grows with prompt length. Prompt caching also lowers input cost on providers that bill cached reads at a discount. Agents, chat applications with long system prompts, and retrieval pipelines that resend the same documents benefit most, since they repeat large identical prefixes on every turn. The cache's prefix rule is why [context engineering](https://theaiengineering.com/articles/context-engineering) treats prompt ordering as a performance decision as well as a quality one, and why [inference optimization](https://theaiengineering.com/articles/llm-inference-optimization) work often starts with cache hit rates.

## Prompt caching vs semantic caching

[Semantic caching](https://theaiengineering.com/glossary/semantic-caching) stores complete responses and returns a stored answer when a new query is similar in meaning, so the model is not called at all. Prompt caching always calls the model and reuses only the input-side computation. A semantic cache can return an answer that does not fit the new query. Prompt caching does not change the output, because a hit requires an exact token match.

## Related terms

- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.

## Guides that cover prompt caching

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [Chunking strategies for RAG: how to choose, tune, and evaluate them](https://theaiengineering.com/articles/chunking-strategies-for-rag): Chunking strategies for RAG compared: fixed-size, recursive, document-aware, semantic, parent-child, and late chunking, with tested Python and an eval method.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.
- [What is retrieval-augmented generation (RAG)? How RAG works](https://theaiengineering.com/articles/what-is-rag): Retrieval-augmented generation (RAG) grounds LLM answers in your documents. Learn the RAG pipeline, hybrid search, agentic RAG, evaluation, and failure modes.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is prompt caching and how does it work?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prompt-caching

- Title: What is prompt caching and how does it work?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/prompt-caching
- Markdown: https://theaiengineering.com/glossary/prompt-caching.md
