---
title: "What is speculative decoding in LLM inference?"
description: "Speculative decoding speeds up LLM inference: a cheap draft proposes tokens and the target model verifies them in one pass, preserving its output distribution."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/speculative-decoding"
markdown_url: "https://theaiengineering.com/glossary/speculative-decoding.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Speculative decoding","speculative sampling","assisted decoding"]
citation: "The AI Engineering Editors. \"What is speculative decoding in LLM inference?\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/speculative-decoding"
---

# What is speculative decoding in LLM inference?

Also called speculative sampling, assisted decoding.

> **Definition of Speculative decoding:** Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.

Speculative decoding is an inference technique that generates text faster by letting a cheap draft guess the next few tokens and having the large target model check all of them in a single forward pass. A large model's decode step is limited mostly by reading its weights from memory, so scoring several candidate positions at once takes about as long as generating one token. When the guesses are right, one target-model step yields several tokens and [inter-token latency](https://theaiengineering.com/glossary/inter-token-latency) falls. With the standard verification rule, the output follows the same distribution the target model would produce on its own.

## How speculative decoding works

Speculative decoding runs a loop of draft, verify, and correct. The draft proposes a short run of tokens, the target model scores every position of that run in one forward pass, and the draft tokens are accepted or rejected in order. Generation continues from the first rejection with a token sampled from the target model, and the loop repeats.

[Leviathan et al.](https://arxiv.org/abs/2211.17192) (ICML 2023) define the acceptance rule. Let `q(x)` be the draft model's probability for a proposed token and `p(x)` the target model's. The token is kept when `q(x) <= p(x)`; otherwise it is rejected with probability `1 - p(x)/q(x)` and the position is resampled from the normalized difference `max(0, p(x) - q(x))`. This rejection-sampling step makes every emitted token follow the target model's distribution. The authors report 2 to 3 times faster decoding on T5-XXL without retraining. [Chen et al.](https://arxiv.org/abs/2302.01318) published the same idea as speculative sampling and report a 2 to 2.5 times speedup on the 70-billion-parameter Chinchilla model in a distributed setup.

The acceptance rate, which the Leviathan paper calls α, measures how often the draft matches the target. With γ draft tokens per step, the expected number of tokens per target pass is `(1 - α^(γ+1)) / (1 - α)`. For example, α = 0.8 with four draft tokens gives about 3.4 tokens per pass, while a poor draft wastes target compute on rejected positions.

## Draft sources

The draft can be any mechanism that proposes likely next tokens more cheaply than the target model. Serving engines support three main families: a separate small model, lightweight heads that predict from the target model's own hidden states, and lookup methods that copy token sequences already present in the prompt.

| Draft source | How it proposes tokens | Trade-off |
|---|---|---|
| Draft model | A smaller model, usually from the target's family, generates the run | Uses extra GPU memory and compute |
| EAGLE heads | A light layer predicts the target's second-to-top-layer features | Needs a head trained for the target model |
| N-gram (prompt lookup) | Matches n-grams in the prompt and proposes what followed them | Almost free; helps when output repeats input |

For EAGLE, [Li et al.](https://arxiv.org/abs/2401.15077) report a 2.7 to 3.5 times latency speedup on LLaMA2-Chat 70B while preserving the output distribution. Hugging Face Transformers calls these methods assisted decoding and suggests prompt lookup for input-grounded tasks such as summarization.

## When speculative decoding helps

Speculative decoding helps most when decode is memory-bound, at low to moderate request rates with small batches. Under heavy load the GPU becomes compute-bound, rejected draft tokens take capacity from other requests, and the gain shrinks. Decode-heavy traffic, such as a [reasoning model](https://theaiengineering.com/glossary/reasoning-model) writing long outputs from short prompts, spends most of its time in the phase speculation shortens.

[vLLM's speculative decoding documentation](https://docs.vllm.ai/en/latest/features/speculative_decoding/) says model-based drafts give the largest latency reductions, while n-gram and suffix methods give modest speedups without adding load at peak traffic. It calls the standard rejection sampler lossless up to the precision limits of hardware numerics and labels some alternative verification modes as lossy. Measure the acceptance rate on your own prompts before enabling speculation for all traffic.

The [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization) compares speculative decoding with batching, caching, and quantization, and [vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama) lists which draft methods each engine supports.

## Related terms

- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.

## Guides that cover speculative decoding

- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is speculative decoding in LLM inference?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/speculative-decoding

- Title: What is speculative decoding in LLM inference?
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/speculative-decoding
- Markdown: https://theaiengineering.com/glossary/speculative-decoding.md
