What is KV cache in LLMs? How it speeds up inference

Also called key-value cache, KV caching, attention cache

Definition

The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.

The KV cache is the store of attention keys and values computed for every token processed so far, kept in GPU memory while a transformer generates text. Each attention layer projects every token into a query, a key, and a value, and each new token's query attends to the keys and values of all earlier tokens. Caching those keys and values means each generation step processes one new token instead of the whole sequence. The KV cache is often the largest consumer of GPU memory after the weights, and at long context lengths or large batch sizes it can exceed them.

How the KV cache works#

Inference runs in two phases. During prefill, the model processes the entire prompt in parallel and writes keys and values for every prompt token into the cache. During decode, it generates one token at a time, computing keys and values only for the newest token, appending them, and reading the rest from the cache.

Three steps: prefill writes cache entries for the four prompt tokens, then each of two decode steps computes one new token and appends one entry. Step and the KV cache after itPrefillComputes 4 prompt tokensThecatsatonDecode 1Computes 1 new token: theThecatsatontheDecode 2Computes 1 new token: matThecatsatonthemat
Figure 1: Prefill fills the cache; each decode step appends one entry.

Prefill time grows with prompt length and is a major component of time to first token. The cache grows by one entry per token until the request finishes or reaches the end of the context window.

Example: how big the KV cache gets#

Per-token KV cache size is 2 (keys and values) × layers × KV heads × head dimension × bytes per value. Llama 3 8B has 32 layers, 8 KV heads, and a head dimension of 128. At 16-bit precision that is 2 × 32 × 8 × 128 × 2 bytes, or 128 KiB per token: 1 GiB for one 8,192-token sequence. A batch of 32 such sequences needs 32 GiB of cache, about twice the roughly 16 GB the model's weights occupy.

That growth explains why so much inference engineering targets the KV cache:

  • Grouped-query attention (Ainslie et al., 2023) shares each key-value head across a group of query heads; Llama 3 8B uses 8 KV heads for 32 query heads, a fourfold reduction.
  • PagedAttention (Kwon et al., 2023), the basis of vLLM, stores the cache in fixed-size blocks to avoid fragmentation and reports 2 to 4 times higher throughput than earlier serving systems at the same latency.
  • KV cache quantization stores keys and values at 8 bits or fewer to fit more tokens in memory.

KV cache vs prompt cache#

Without prefix reuse, a request's KV cache is freed when the request finishes. Prompt caching keeps the KV cache for a shared prefix after the request ends and reuses it for later requests that begin with the same tokens. For other ways to reduce inference memory and latency, see the LLM inference optimization guide.

Cite this definition

The AI Engineering Editors. "What is KV cache in LLMs? How it speeds up inference." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/kv-cache

Title
What is KV cache in LLMs? How it speeds up inference
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM inference and serving hub · Updated