---
title: "What is KV cache in LLMs? How it speeds up inference"
description: "The KV cache stores attention keys and values for past tokens so an LLM skips recomputing them at each step. How it works, how big it gets, and vs prompt cache."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/kv-cache"
markdown_url: "https://theaiengineering.com/glossary/kv-cache.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["KV cache","key-value cache","KV caching","attention cache"]
citation: "The AI Engineering Editors. \"What is KV cache in LLMs? How it speeds up inference.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/kv-cache"
---

# What is KV cache in LLMs? How it speeds up inference

Also called key-value cache, KV caching, attention cache.

> **Definition of KV cache:** The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.

The KV cache is the store of attention keys and values computed for every token processed so far, kept in GPU memory while a [transformer](https://theaiengineering.com/glossary/transformer) generates text. Each attention layer projects every token into a query, a key, and a value, and each new token's query attends to the keys and values of all earlier tokens. Caching those keys and values means each generation step processes one new token instead of the whole sequence. The KV cache is often the largest consumer of GPU memory after the weights, and at long context lengths or large batch sizes it can exceed them.

## How the KV cache works

Inference runs in two phases. During [prefill](https://theaiengineering.com/glossary/prefill-and-decode), the model processes the entire prompt in parallel and writes keys and values for every prompt token into the cache. During decode, it generates one token at a time, computing keys and values only for the newest token, appending them, and reading the rest from the cache.

**Figure 1: Prefill fills the cache; each decode step appends one entry.**

Three steps: prefill writes cache entries for the four prompt tokens, then each of two decode steps computes one new token and appends one entry.

Layers, top to bottom:

1. **Prefill**: Computes 4 prompt tokens (The, cat, sat, on)
2. **Decode 1**: Computes 1 new token: the (The, cat, sat, on, the)
3. **Decode 2**: Computes 1 new token: mat (The, cat, sat, on, the, mat)

Prefill time grows with prompt length and is a major component of [time to first token](https://theaiengineering.com/glossary/time-to-first-token). The cache grows by one entry per token until the request finishes or reaches the end of the [context window](https://theaiengineering.com/glossary/context-window).

## Example: how big the KV cache gets

Per-token KV cache size is 2 (keys and values) × layers × KV heads × head dimension × bytes per value. Llama 3 8B has 32 layers, 8 KV heads, and a head dimension of 128. At 16-bit precision that is 2 × 32 × 8 × 128 × 2 bytes, or 128 KiB per token: 1 GiB for one 8,192-token sequence. A batch of 32 such sequences needs 32 GiB of cache, about twice the roughly 16 GB the model's weights occupy.

That growth explains why so much inference engineering targets the KV cache:

- **Grouped-query attention** ([Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)) shares each key-value head across a group of query heads; Llama 3 8B uses 8 KV heads for 32 query heads, a fourfold reduction.
- **[PagedAttention](https://theaiengineering.com/glossary/paged-attention)** ([Kwon et al., 2023](https://arxiv.org/abs/2309.06180)), the basis of vLLM, stores the cache in fixed-size blocks to avoid fragmentation and reports 2 to 4 times higher throughput than earlier serving systems at the same latency.
- **KV cache [quantization](https://theaiengineering.com/glossary/quantization)** stores keys and values at 8 bits or fewer to fit more tokens in memory.

## KV cache vs prompt cache

Without prefix reuse, a request's KV cache is freed when the request finishes. [Prompt caching](https://theaiengineering.com/glossary/prompt-caching) keeps the KV cache for a shared prefix after the request ends and reuses it for later requests that begin with the same tokens. For other ways to reduce inference memory and latency, see the [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization).

## Related terms

- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.

## Guides that cover kv cache

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [Semantic caching for LLMs: how it works and when it pays off](https://theaiengineering.com/articles/semantic-caching): Semantic caching returns a stored LLM response for prompts with similar meaning. Learn how a semantic cache works, how to tune it, and what never to cache.
- [What is context engineering? A practical guide for AI agents](https://theaiengineering.com/articles/context-engineering): Context engineering is the practice of choosing what fills an LLM context window. Learn its components, why context rot happens, and how to measure results.
- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is KV cache in LLMs? How it speeds up inference." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/kv-cache

- Title: What is KV cache in LLMs? How it speeds up inference
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/kv-cache
- Markdown: https://theaiengineering.com/glossary/kv-cache.md
