What is KV cache in LLMs? How it speeds up inference
Also called key-value cache, KV caching, attention cache
Definition
The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
The KV cache is the store of attention keys and values computed for every token processed so far, kept in GPU memory while a transformer generates text. Each attention layer projects every token into a query, a key, and a value, and each new token's query attends to the keys and values of all earlier tokens. Caching those keys and values means each generation step processes one new token instead of the whole sequence. The KV cache is often the largest consumer of GPU memory after the weights, and at long context lengths or large batch sizes it can exceed them.
How the KV cache works#
Inference runs in two phases. During prefill, the model processes the entire prompt in parallel and writes keys and values for every prompt token into the cache. During decode, it generates one token at a time, computing keys and values only for the newest token, appending them, and reading the rest from the cache.
Prefill time grows with prompt length and is a major component of time to first token. The cache grows by one entry per token until the request finishes or reaches the end of the context window.
Example: how big the KV cache gets#
Per-token KV cache size is 2 (keys and values) × layers × KV heads × head dimension × bytes per value. Llama 3 8B has 32 layers, 8 KV heads, and a head dimension of 128. At 16-bit precision that is 2 × 32 × 8 × 128 × 2 bytes, or 128 KiB per token: 1 GiB for one 8,192-token sequence. A batch of 32 such sequences needs 32 GiB of cache, about twice the roughly 16 GB the model's weights occupy.
That growth explains why so much inference engineering targets the KV cache:
- Grouped-query attention (Ainslie et al., 2023) shares each key-value head across a group of query heads; Llama 3 8B uses 8 KV heads for 32 query heads, a fourfold reduction.
- PagedAttention (Kwon et al., 2023), the basis of vLLM, stores the cache in fixed-size blocks to avoid fragmentation and reports 2 to 4 times higher throughput than earlier serving systems at the same latency.
- KV cache quantization stores keys and values at 8 bits or fewer to fit more tokens in memory.
KV cache vs prompt cache#
Without prefix reuse, a request's KV cache is freed when the request finishes. Prompt caching keeps the KV cache for a shared prefix after the request ends and reuses it for later requests that begin with the same tokens. For other ways to reduce inference memory and latency, see the LLM inference optimization guide.
Related terms
Guides that cover kv cache
- LLM inference optimization: a practical guide for engineers
- vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use
- Semantic caching for LLMs: how it works and when it pays off
- What is context engineering? A practical guide for AI agents
- LLM routing: strategies, fallbacks, and learned routers
- RAG vs fine-tuning: how to choose for your LLM application
Cite this definition
The AI Engineering Editors. "What is KV cache in LLMs? How it speeds up inference." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/kv-cache
- Title
- What is KV cache in LLMs? How it speeds up inference
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated