What are prefill and decode in LLM inference?
Also called prefill phase, decode phase, prefill vs decode, context and generation phases
Definition
Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
Prefill and decode are the two phases every request to a decoder-only transformer goes through. Prefill runs the full prompt through the model in parallel, writes keys and values for every prompt token into the KV cache, and produces the first output token. Decode then generates the remaining tokens one forward pass at a time, reading the cache at each step. The two phases hit different hardware limits, so they drive different latency metrics and respond to different optimizations. NVIDIA's TensorRT-LLM calls them the context phase and the generation phase.
Prefill vs decode#
Prefill is compute-bound and decode is memory-bandwidth-bound. Prefill multiplies large matrices over many prompt tokens at once, so the GPU's arithmetic throughput limits it. Decode handles one new token per sequence per step but still reads all model weights and the sequence's cached keys and values, so the speed of GPU memory limits it.
The DistServe paper (Zhong et al., OSDI 2024) describes prefill as compute-bound for prompts that are not brief and a single decoding job as heavily bandwidth-bound. Batching therefore matters more for decode, where many sequences share each read of the weights, than for prefill, where adding requests stops helping once the GPU is compute-bound.
| Prefill | Decode | |
|---|---|---|
| Tokens per forward pass | Every prompt token | One per sequence |
| Main bottleneck | GPU compute | GPU memory bandwidth |
| Latency metric it drives | Time to first token | Inter-token latency |
| Work grows with | Prompt length | Output length |
Queueing plus prefill make up most of time to first token, and decode step time sets inter-token latency. Workloads lean one way or the other. A retrieval-augmented request that adds long documents to the prompt is prefill-heavy, so it gains from tighter context and from prompt caching of any repeated prefix, such as a long system prompt.
A reasoning model that writes thousands of tokens from a short prompt is decode-heavy, and speculative decoding helps it by emitting several tokens per target-model step. The reasoning models guide explains why that traffic is decode-heavy.
Chunked prefill and disaggregation#
When prefill and decode share a GPU, a long prompt can stall every in-flight decode while its prefill runs. Engines handle this interference in two ways: chunked prefill splits long prompts into pieces that are mixed into decode steps, and prefill-decode disaggregation runs the two phases on separate GPU pools and moves the KV cache between them.
Sarathi-Serve (Agrawal et al., 2024) splits each prefill into near-equal chunks and builds stall-free schedules that add new requests to a batch without pausing ongoing decodes. vLLM's optimization guide says chunked prefill is enabled by default whenever possible; the scheduler batches all pending decodes first, then fills the remaining max_num_batched_tokens budget with prefill chunks. A smaller budget favors inter-token latency, and a larger one favors time to first token and throughput.
DistServe takes the other route. It runs prefill and decode on different GPUs, sends the KV cache from prefill instances to decode instances, and chooses resources and parallelism separately for each phase. The authors report serving 7.4 times more requests, or meeting 12.6 times tighter latency objectives, than the systems they compared against. The cost is a fast network for KV transfer and a more complex deployment.
The LLM inference optimization guide maps each serving technique to the phase it speeds up.
Related terms
Guides that cover prefill and decode
Cite this definition
The AI Engineering Editors. "What are prefill and decode in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prefill-and-decode
- Title
- What are prefill and decode in LLM inference?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated