What is PagedAttention? How vLLM manages KV cache
Also called paged attention, paged KV cache
Definition
PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
PagedAttention is the KV cache memory scheme that Kwon et al. introduced with the vLLM serving engine at SOSP 2023, describing it as an attention algorithm inspired by virtual memory and paging in operating systems. It splits each request's KV cache into fixed-size blocks, allocates a new block only when the previous one is full, and lets the attention kernel read keys and values from blocks scattered across GPU memory. Less wasted memory means more sequences fit in each batch. The authors report 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency.
How PagedAttention works#
PagedAttention treats the KV cache like virtual memory. Each sequence sees logical blocks filled left to right, and a block table maps every logical block to a physical block somewhere in GPU memory and records how many slots are filled. The attention kernel follows the table to fetch each block, so blocks need not be contiguous.
The paper maps the analogy directly, treating blocks as pages, tokens as bytes, and requests as processes. Each block holds the keys and values for a fixed number of tokens. vLLM's default block size in the paper is 16 tokens, chosen to be large enough for efficient GPU kernels and small enough to keep internal fragmentation low. A request's only unused memory is the unfilled tail of its last block. A 1,000-token sequence, for example, occupies 63 blocks of 16 slots and leaves 8 slots empty.
PagedAttention vs contiguous allocation#
Serving systems before PagedAttention reserved one contiguous memory region per request, sized for the maximum possible output. That design wastes memory three ways: slots reserved for tokens the request has not generated yet, internal fragmentation when the output ends early, and external fragmentation in the gaps between regions of different sizes.
The vLLM paper measured that only 20.4% to 38.2% of KV cache memory held actual token states in the existing systems it profiled. Paging removes most of that waste:
| Contiguous allocation | PagedAttention | |
|---|---|---|
| Allocation unit | One region per request, sized for the maximum length | One fixed-size block, allocated on demand |
| Reserved but unused memory | Up to the full maximum output length | At most the unfilled part of the last block |
| External fragmentation | Gaps between regions of different sizes | None, because all blocks are the same size |
| Sharing between requests | Each request holds its own copy | Several block tables can point at one physical block |
When a request finishes, its blocks return to a shared pool, so continuous batching can admit a queued request as soon as enough blocks are free.
Prefix sharing and copy-on-write#
Because a block table is only a mapping, two sequences can point at the same physical block. PagedAttention uses this to share one copy of a prompt's KV cache across parallel samples, beam search candidates, and requests with the same system prompt, and copies a shared block only when one sequence must write to it.
Reference counts track how many sequences use each block, and the paper implements copy-on-write at block granularity, so diverging samples copy one block instead of the whole cache. It also proposes reserving blocks for predefined shared prefixes so that a request's prefill computes only the user's own input. Current vLLM generalizes this as automatic prefix caching: its prefix caching design hashes each full block together with its parent block's hash, reuses matching blocks across requests, and evicts unused blocks in least-recently-used order. Hosted APIs sell the same effect as prompt caching. vLLM's kernels have been rewritten since the paper, but block-based KV cache management remains, and other engines such as TensorRT-LLM also document a paged KV cache.
The LLM inference optimization guide shows how PagedAttention fits with batching, prefix caching, and quantization.
Related terms
Guides that cover pagedattention
Cite this definition
The AI Engineering Editors. "What is PagedAttention? How vLLM manages KV cache." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/paged-attention
- Title
- What is PagedAttention? How vLLM manages KV cache
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated