What is speculative decoding in LLM inference?
Also called speculative sampling, assisted decoding
Definition
Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
Speculative decoding is an inference technique that generates text faster by letting a cheap draft guess the next few tokens and having the large target model check all of them in a single forward pass. A large model's decode step is limited mostly by reading its weights from memory, so scoring several candidate positions at once takes about as long as generating one token. When the guesses are right, one target-model step yields several tokens and inter-token latency falls. With the standard verification rule, the output follows the same distribution the target model would produce on its own.
How speculative decoding works#
Speculative decoding runs a loop of draft, verify, and correct. The draft proposes a short run of tokens, the target model scores every position of that run in one forward pass, and the draft tokens are accepted or rejected in order. Generation continues from the first rejection with a token sampled from the target model, and the loop repeats.
Leviathan et al. (ICML 2023) define the acceptance rule. Let q(x) be the draft model's probability for a proposed token and p(x) the target model's. The token is kept when q(x) <= p(x); otherwise it is rejected with probability 1 - p(x)/q(x) and the position is resampled from the normalized difference max(0, p(x) - q(x)). This rejection-sampling step makes every emitted token follow the target model's distribution. The authors report 2 to 3 times faster decoding on T5-XXL without retraining. Chen et al. published the same idea as speculative sampling and report a 2 to 2.5 times speedup on the 70-billion-parameter Chinchilla model in a distributed setup.
The acceptance rate, which the Leviathan paper calls α, measures how often the draft matches the target. With γ draft tokens per step, the expected number of tokens per target pass is (1 - α^(γ+1)) / (1 - α). For example, α = 0.8 with four draft tokens gives about 3.4 tokens per pass, while a poor draft wastes target compute on rejected positions.
Draft sources#
The draft can be any mechanism that proposes likely next tokens more cheaply than the target model. Serving engines support three main families: a separate small model, lightweight heads that predict from the target model's own hidden states, and lookup methods that copy token sequences already present in the prompt.
| Draft source | How it proposes tokens | Trade-off |
|---|---|---|
| Draft model | A smaller model, usually from the target's family, generates the run | Uses extra GPU memory and compute |
| EAGLE heads | A light layer predicts the target's second-to-top-layer features | Needs a head trained for the target model |
| N-gram (prompt lookup) | Matches n-grams in the prompt and proposes what followed them | Almost free; helps when output repeats input |
For EAGLE, Li et al. report a 2.7 to 3.5 times latency speedup on LLaMA2-Chat 70B while preserving the output distribution. Hugging Face Transformers calls these methods assisted decoding and suggests prompt lookup for input-grounded tasks such as summarization.
When speculative decoding helps#
Speculative decoding helps most when decode is memory-bound, at low to moderate request rates with small batches. Under heavy load the GPU becomes compute-bound, rejected draft tokens take capacity from other requests, and the gain shrinks. Decode-heavy traffic, such as a reasoning model writing long outputs from short prompts, spends most of its time in the phase speculation shortens.
vLLM's speculative decoding documentation says model-based drafts give the largest latency reductions, while n-gram and suffix methods give modest speedups without adding load at peak traffic. It calls the standard rejection sampler lossless up to the precision limits of hardware numerics and labels some alternative verification modes as lossy. Measure the acceptance rate on your own prompts before enabling speculation for all traffic.
The LLM inference optimization guide compares speculative decoding with batching, caching, and quantization, and vLLM vs Ollama lists which draft methods each engine supports.
Related terms
Guides that cover speculative decoding
Cite this definition
The AI Engineering Editors. "What is speculative decoding in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/speculative-decoding
- Title
- What is speculative decoding in LLM inference?
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated