LLM inference optimization: a practical guide for engineers
LLM inference optimization is the set of serving techniques that reduce the latency, memory footprint, and cost of generating tokens from a large language model while keeping output quality close to an unoptimized baseline. Most production gains come from a short list of methods: managing the KV cache efficiently, batching at the level of individual decode steps, quantizing weights, decoding speculatively, reusing shared prefixes, and splitting models across GPUs. This guide explains why each method works, which metric each one improves, and what to try first. It is part of the LLM inference topic hub.
What is LLM inference?#
LLM inference is running a trained model on a prompt to generate output tokens one at a time, each conditioned on the prompt and all earlier tokens. Training happens once, but inference runs for every request a product serves, so inference efficiency sets both the serving bill and the latency users feel.
A decoder-only transformer produces one token per forward pass. Every request moves through two phases with different hardware bottlenecks, and most optimizations target one phase or the boundary between them.
Prefill vs decode#
- Prefill processes all prompt tokens in a single parallel forward pass, computes their attention keys and values, and emits the first output token. The work is dominated by large matrix multiplications, so prefill is compute-bound. Prefill time grows with prompt length and dominates time to first token.
- Decode generates one token per forward pass. Each step reads the full set of model weights and the request's cached keys and values from GPU memory but performs little arithmetic per byte read, so decode is limited by memory bandwidth at typical batch sizes. Decode step time sets inter-token latency.
Which metrics matter for LLM inference?#
Four numbers describe most serving behavior: time to first token, inter-token latency, end-to-end latency, and throughput. Latency metrics describe one user's experience, while throughput describes the capacity of the server. Larger batches usually raise throughput and also raise per-request latency, so judge every optimization against an explicit latency target.
| Metric | What it measures | Dominated by | Watch out for |
|---|---|---|---|
| Time to first token (TTFT) | Time from request arrival to the first output token | Queueing plus prefill of the prompt | Long prompts and prefill contention under load |
| Inter-token latency (ITL), also time per output token (TPOT) | Average gap between output tokens after the first | Decode step time at the current batch size | Large prefills scheduled alongside decodes |
| End-to-end latency | Total time to finish a response | Roughly TTFT plus ITL times the number of output tokens | Long outputs, including hidden reasoning tokens |
| Output throughput | Generated tokens per second across all requests | Batch size, memory bandwidth, KV cache capacity | Peak numbers measured without a latency limit |
| Request throughput | Completed requests per second | The mix of prompt and output lengths | Comparisons across different workloads |
| Goodput | Highest request rate that still meets latency targets | All of the above | The metric that maps most directly to cost per query |
| KV cache usage and preemptions | Share of cache blocks in use; requests evicted and recomputed | Concurrency and context length | Preemptions that inflate end-to-end latency |
Report latency as percentiles (p50, p95, p99) at a fixed request rate rather than as a single average. The DistServe paper on disaggregated serving uses goodput as its central metric, defined as the maximum request rate a deployment can serve while meeting its latency objectives. In production, the same metrics belong in your LLM observability dashboards.
How does the KV cache work?#
The KV cache stores the attention key and value vectors computed for every previous token, so each decode step only computes vectors for the newest token instead of reprocessing the whole sequence. The cache grows linearly with sequence length and with the number of concurrent sequences, and it competes with model weights for GPU memory.
The per-token size follows directly from the model configuration:
KV bytes per token = 2 (K and V) × layers × KV heads × head dim × bytesLlama 3 70B has 80 layers, 8 key-value heads, and a head dimension of 128. At 16-bit precision (2 bytes per value), that is 327,680 bytes, or 320 KiB per token. One 8,192-token sequence needs 2.5 GiB of cache, and 32 such sequences need 80 GiB, on top of roughly 140 GB of 16-bit weights. The vLLM authors estimate 800 KB per token for OPT-13B, a smaller model that uses full multi-head attention without key-value head sharing.
Three levers shrink the cache:
- Grouped-query attention (GQA) shares each key-value head across several query heads. Llama 3 70B uses 8 key-value heads for 64 query heads, so its cache is one eighth the size full multi-head attention would need.
- KV cache quantization stores keys and values in 8 bits or fewer. vLLM, llama.cpp, and Ollama all document quantized KV cache options.
- Context limits cap the maximum sequence length per request, which bounds the worst-case cache per sequence.
PagedAttention and memory fragmentation#
Early serving systems reserved a contiguous memory region per request, sized for the maximum possible output. The vLLM paper on PagedAttention (Kwon et al., SOSP 2023) measured that only 20.4% to 38.2% of KV cache memory held actual token states in the existing systems it studied; reservation and fragmentation consumed the rest.
PagedAttention applies operating-system paging to the cache. The engine splits each sequence's cache into fixed-size blocks, allocates blocks on demand, stores them anywhere in GPU memory, and lets requests share identical blocks. The authors report 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency. Block-based KV management is now standard: TensorRT-LLM documents a paged KV cache, and SGLang's RadixAttention is designed to work alongside paged attention.
What is continuous batching?#
Continuous batching schedules work one decode iteration at a time rather than one request at a time, so a finished sequence leaves the batch immediately and a waiting request joins at the next step. Output lengths vary widely between requests, and iteration-level scheduling keeps the GPU busy instead of idling until the longest sequence in a static batch finishes.
The idea was introduced as iteration-level scheduling in Orca, presented at OSDI 2022, which paired it with selective batching of operations that tolerate sequences of different lengths. vLLM, SGLang, TensorRT-LLM (where the feature is called in-flight batching), and llama.cpp's server all implement continuous batching today. Continuous batching also explains why per-request benchmarks mislead: a single request on an idle server measures latency, not the throughput the same server delivers at 50 concurrent users.
Chunked prefill#
A long prompt arriving mid-stream can stall every in-flight decode while its prefill runs. Chunked prefill splits large prefills into pieces and mixes them with decode steps. In vLLM's scheduler, pending decodes are batched first, and the remaining per-step token budget (max_num_batched_tokens) is filled with prefill chunks. vLLM's documentation describes the trade-off: a smaller budget improves inter-token latency, and a larger budget improves time to first token and throughput.
LLM quantization: what to quantize and what it costs#
Quantization stores weights, activations, or the KV cache in fewer bits than the 16-bit format most models ship in. Fewer bits mean less memory for the model and fewer bytes to read per decode step, which raises the batch a GPU can hold and speeds up memory-bound decoding. Lower precision can also reduce accuracy, so evaluate every quantized model.
| Scheme | What is quantized | Example support | Hardware note |
|---|---|---|---|
| INT4 weight-only (GPTQ, AWQ) | Weights only; activations stay 16-bit | vLLM | Gains are largest when decode is memory-bound, typically at small batch sizes |
| FP8 (W8A8) | Weights and activations | vLLM, TensorRT-LLM | vLLM documents official W8A8 support on Ada Lovelace and Hopper GPUs (compute capability 8.9 and above) |
| FP4 (NVFP4, MXFP4) | Weights, often activations | vLLM, TensorRT-LLM | NVIDIA introduced NVFP4 with the Blackwell architecture |
| GGUF k-quants (for example Q4_K_M) | Weights, in 1.5-bit to 8-bit integer formats | llama.cpp, Ollama, LM Studio | Runs on CPUs, Apple silicon, and consumer GPUs |
| 8-bit KV cache | Cached keys and values | vLLM, llama.cpp, Ollama | Doubles cache capacity relative to 16-bit; check long-context quality |
The GPTQ paper (Frantar et al., ICLR 2023) reports quantizing GPT-family models to 3 or 4 bits per weight with little accuracy loss on its benchmarks. Treat published results as a starting point, not a guarantee: re-run your task-specific suite from your LLM evaluation process after quantizing, because regressions often show up in formatting, rare languages, or long-context retrieval before they show up in aggregate scores.
What is speculative decoding?#
Speculative decoding uses a cheap draft mechanism to propose several tokens, then runs the target model once to verify all of them and keeps the longest accepted run. Because decode steps are memory-bound, verifying several tokens costs about the same as generating one, so each accepted draft token reduces inter-token latency.
Fast Inference from Transformers via Speculative Decoding (Leviathan et al., ICML 2023) showed that a rejection-sampling verification step makes the output distribution identical to the target model's, and reported 2 to 3 times faster decoding on T5-XXL. Current engines offer several draft sources:
- Draft models: a smaller model from the same family with a shared vocabulary.
- EAGLE and multi-token prediction (MTP) heads: light prediction layers attached to the target model.
- N-gram and suffix matching: proposals copied from text already in the prompt or earlier output, which suit code edits and extraction tasks that repeat input text.
vLLM's documentation targets speculative decoding at medium-to-low request rates, where decode is memory-bound. Under heavy load the GPU becomes compute-bound, rejected drafts waste capacity, and the speedup shrinks. Measure the acceptance rate on your own traffic before enabling it everywhere.
What is prefix caching?#
Prefix caching keeps the KV cache blocks of completed requests and reuses them when a new request begins with the same tokens, which skips prefill for the shared part. Long system prompts, few-shot examples, multi-turn chat history, and agent tool definitions all benefit, because every turn resends the same prefix.
vLLM's automatic prefix caching hashes each full cache block together with its parent block's hash, so a block matches only when the entire preceding prefix matches. vLLM can mix a per-request cache salt into the hash to keep tenants from sharing cache entries. SGLang stores prefixes in a radix tree (RadixAttention) with least-recently-used eviction, and llama.cpp's server reuses the KV cache from a previous request through its cache_prompt option.
Hosted APIs offer a comparable feature called prompt caching, usually with discounted pricing for cached input tokens. Prefix caching is different from semantic caching, which returns a stored response for a similar query and usually runs at an AI gateway.
To raise hit rates, place stable content (instructions, tools, reference documents) first and variable content last, a habit covered in context engineering.
Tensor parallelism vs pipeline parallelism#
Tensor parallelism splits each layer's weight matrices across GPUs, so every GPU works on every token and the GPUs exchange activations inside each layer. Pipeline parallelism assigns consecutive groups of layers to different GPUs or nodes, which needs less communication but leaves stages idle while the pipeline fills. Use tensor parallelism inside a node and add pipeline parallelism across nodes.
vLLM's scaling guidance maps cleanly to deployment size:
- Model fits on one GPU: no model parallelism; add replicas (data parallelism) for throughput.
- Model fits on one node: set
--tensor-parallel-sizeto the number of GPUs in the node. - Model needs several nodes: tensor parallelism within each node, and
--pipeline-parallel-sizeequal to the number of nodes. - GPUs without NVLink: vLLM recommends pipeline parallelism, because tensor parallelism's per-layer communication saturates slower links.
Mixture-of-experts models add expert parallelism, which places different experts on different GPUs and routes tokens between them.
Prefill-decode disaggregation#
Prefill and decode compete when they share GPUs: a burst of long prompts slows every in-flight decode. Disaggregated serving runs the two phases on separate GPU pools and transfers the KV cache between them. DistServe reported serving up to 7.4 times more requests, or meeting 12.6 times tighter latency targets, than colocated baselines. vLLM, SGLang, and TensorRT-LLM (in beta) now support disaggregated prefill and decode; the cost is a fast network for KV transfer and a more complex deployment.
How to choose a GPU for LLM inference#
Choose hardware by four properties, in order: memory capacity for weights plus KV cache at your target concurrency, memory bandwidth for decode speed, supported low-precision formats such as FP8 and FP4, and GPU-to-GPU interconnect if the model needs tensor parallelism. Peak compute matters most for prefill-heavy work such as long-document processing.
A sizing pass takes three steps:
- Weights: parameters times bytes per parameter. A 70B model needs about 140 GB at 16-bit, about 70 GB at FP8, and about 35 GB at 4-bit, plus quantization scales and runtime overhead.
- KV cache: per-token size (from the formula above) times context length times concurrent sequences.
- Decode ceiling: at batch size 1, each token requires reading all weights once, so memory bandwidth divided by weight bytes approximates the upper bound on single-stream tokens per second. Batching raises total throughput by sharing that read across sequences.
Apple silicon Macs and consumer GPUs offer less memory bandwidth than data center GPUs at a lower cost, and Apple's unified memory architecture shares one memory pool between CPU and GPU, which suits local development; the vLLM vs Ollama comparison covers engines built for those machines. If operating GPUs is not a requirement, hosted inference providers sell the same optimizations as a per-token service.
Which optimization should you try first?#
Start with changes that modern engines provide at no quality cost, then move to changes that alter the model or the deployment topology. A practical order: adopt an engine with paged KV and continuous batching, enable prefix caching, tune batch limits to your latency target, quantize, add speculative decoding, and only then scale out with parallelism or disaggregation.
| Technique | Primarily improves | Main cost or risk | Changes outputs? |
|---|---|---|---|
| Paged KV cache | Concurrency, throughput | None in engines that ship it | No |
| Continuous batching | Throughput, queueing delay | Higher ITL at very large batches | No |
| Chunked prefill | ITL stability under load | Tuning the per-step token budget | No |
| Prefix caching | TTFT, prefill compute | Memory held by cached blocks; tenant isolation | No |
| Weight quantization | Memory, decode speed, cost | Possible quality loss; kernel support varies | Can |
| KV cache quantization | Concurrency, context length | Possible long-context quality loss | Can |
| Speculative decoding | ITL at low-to-medium load | Draft overhead; gains shrink at high load | No, with standard rejection sampling |
| Tensor or pipeline parallelism | Fits larger models; per-token latency | Interconnect bandwidth, idle pipeline stages | No |
| Prefill-decode disaggregation | Goodput under strict TTFT and ITL targets | KV transfer network, operational complexity | No |
"No" in the last column means the technique preserves the model's output distribution; results can still differ at the floating-point level across batch sizes and kernels.
To pick an engine that implements these techniques, read the vLLM vs Ollama comparison and browse the serving engines directory. If your traffic includes long chains of hidden reasoning, reasoning models explains why those workloads are decode-heavy and how effort settings change their cost.
Frequently asked questions
What is prefill and decode in LLM inference?
- Prefill is the first phase of a request: the model processes every prompt token in one parallel forward pass, fills the KV cache, and emits the first output token. Decode is the second phase: the model generates one token per forward pass, reading all weights and the cached keys and values each step. Prefill is compute-bound and drives time to first token; decode is memory-bound and drives inter-token latency.
How do you speed up LLM inference?
- Start with a serving engine that implements paged KV cache management and continuous batching, such as vLLM, SGLang, or TensorRT-LLM. Then enable prefix caching for repeated prompt prefixes, tune batch token limits for your latency target, quantize weights to FP8 or 4-bit formats after checking quality, and add speculative decoding for latency at low to medium load. Use tensor parallelism when the model does not fit on one GPU.
What does the KV cache store?
- The KV cache stores the key and value vectors that each attention layer computed for every previous token in a sequence. Reusing those vectors means each decode step only computes keys and values for the newest token. Cache size per token equals two times the number of layers, key-value heads, head dimension, and bytes per value, so long contexts and many concurrent requests consume GPU memory quickly.
What is a good time to first token?
- No universal number applies, because acceptable time to first token depends on the product. Streaming chat interfaces need the first token fast enough that the interface feels responsive, while batch summarization jobs can tolerate seconds. Set a target per route from user research or product requirements, measure p50 and p95 under realistic load, and remember that long prompts, queueing, and hidden reasoning tokens all delay the first visible token.
Does LLM quantization reduce quality?
- LLM quantization can reduce quality, and the size of the loss depends on the format, the method, the model, and the task. Published methods such as GPTQ report small accuracy losses at 3 or 4 bits per weight on the benchmarks they test, and 8-bit formats discard less information than 4-bit formats. Run your own evaluation set on the quantized model before shipping, because aggregate benchmark scores can hide task-specific regressions.
What is the best GPU for LLM inference?
- The best GPU for LLM inference is the one whose memory holds the model weights plus the KV cache for your target concurrency, with enough memory bandwidth for your decode latency target. After that, check support for the precision formats you plan to use, such as FP8 or FP4, and the interconnect between GPUs if the model needs tensor parallelism. Benchmark your actual model and traffic before buying or reserving capacity.
Terms in this guide
- KV cache
- Time to first token
- Quantization
- Prompt caching
- Semantic caching
- Mixture of experts
- Tokens
- Large language model (LLM)
- Transformer
- Prefill and decode
- Inter-token latency
- Inference engine
- PagedAttention
- Continuous batching
- GGUF
- Speculative decoding
- System prompt
- Few-shot prompting
- Tensor parallelism
- Test-time compute
Cite this guide
The AI Engineering Editors. "LLM inference optimization: a practical guide for engineers." The AI Engineering, updated Oct 23, 2026. https://theaiengineering.com/articles/llm-inference-optimization
- Title
- LLM inference optimization: a practical guide for engineers
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Published
- Updated