---
title: "What are prefill and decode in LLM inference?"
description: "Prefill and decode are the two phases of LLM inference: a compute-bound pass over the prompt, then memory-bound token generation. How each phase shapes latency."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/prefill-and-decode"
markdown_url: "https://theaiengineering.com/glossary/prefill-and-decode.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Prefill and decode","prefill phase","decode phase","prefill vs decode","context and generation phases"]
citation: "The AI Engineering Editors. \"What are prefill and decode in LLM inference?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prefill-and-decode"
---

# What are prefill and decode in LLM inference?

Also called prefill phase, decode phase, prefill vs decode, context and generation phases.

> **Definition of Prefill and decode:** Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.

Prefill and decode are the two phases every request to a decoder-only [transformer](https://theaiengineering.com/glossary/transformer) goes through. Prefill runs the full prompt through the model in parallel, writes keys and values for every prompt token into the [KV cache](https://theaiengineering.com/glossary/kv-cache), and produces the first output token. Decode then generates the remaining tokens one forward pass at a time, reading the cache at each step. The two phases hit different hardware limits, so they drive different latency metrics and respond to different optimizations. NVIDIA's TensorRT-LLM calls them the context phase and the generation phase.

## Prefill vs decode

Prefill is compute-bound and decode is memory-bandwidth-bound. Prefill multiplies large matrices over many prompt tokens at once, so the GPU's arithmetic throughput limits it. Decode handles one new token per sequence per step but still reads all model weights and the sequence's cached keys and values, so the speed of GPU memory limits it.

The [DistServe paper](https://arxiv.org/abs/2401.09670) (Zhong et al., OSDI 2024) describes prefill as compute-bound for prompts that are not brief and a single decoding job as heavily bandwidth-bound. Batching therefore matters more for decode, where many sequences share each read of the weights, than for prefill, where adding requests stops helping once the GPU is compute-bound.

| | Prefill | Decode |
|---|---|---|
| Tokens per forward pass | Every prompt token | One per sequence |
| Main bottleneck | GPU compute | GPU memory bandwidth |
| Latency metric it drives | Time to first token | Inter-token latency |
| Work grows with | Prompt length | Output length |

Queueing plus prefill make up most of [time to first token](https://theaiengineering.com/glossary/time-to-first-token), and decode step time sets [inter-token latency](https://theaiengineering.com/glossary/inter-token-latency). Workloads lean one way or the other. A retrieval-augmented request that adds long documents to the prompt is prefill-heavy, so it gains from tighter context and from [prompt caching](https://theaiengineering.com/glossary/prompt-caching) of any repeated prefix, such as a long system prompt.

A reasoning model that writes thousands of tokens from a short prompt is decode-heavy, and [speculative decoding](https://theaiengineering.com/glossary/speculative-decoding) helps it by emitting several tokens per target-model step. The [reasoning models guide](https://theaiengineering.com/articles/reasoning-models) explains why that traffic is decode-heavy.

## Chunked prefill and disaggregation

When prefill and decode share a GPU, a long prompt can stall every in-flight decode while its prefill runs. Engines handle this interference in two ways: chunked prefill splits long prompts into pieces that are mixed into decode steps, and prefill-decode disaggregation runs the two phases on separate GPU pools and moves the KV cache between them.

[Sarathi-Serve](https://arxiv.org/abs/2403.02310) (Agrawal et al., 2024) splits each prefill into near-equal chunks and builds stall-free schedules that add new requests to a batch without pausing ongoing decodes. [vLLM's optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/) says chunked prefill is enabled by default whenever possible; the scheduler batches all pending decodes first, then fills the remaining `max_num_batched_tokens` budget with prefill chunks. A smaller budget favors inter-token latency, and a larger one favors time to first token and throughput.

DistServe takes the other route. It runs prefill and decode on different GPUs, sends the KV cache from prefill instances to decode instances, and chooses resources and parallelism separately for each phase. The authors report serving 7.4 times more requests, or meeting 12.6 times tighter latency objectives, than the systems they compared against. The cost is a fast network for KV transfer and a more complex deployment.

The [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) maps each serving technique to the phase it speeds up.

## Related terms

- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.

## Guides that cover prefill and decode

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What are prefill and decode in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/prefill-and-decode

- Title: What are prefill and decode in LLM inference?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/prefill-and-decode
- Markdown: https://theaiengineering.com/glossary/prefill-and-decode.md
