---
title: "LLM inference optimization: a practical guide for engineers"
description: "LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs."
summary: "LLM inference optimization is the set of serving techniques that cut latency, memory use, and cost per token without retraining the model. This guide explains prefill and decode, the metrics worth tracking, and which technique moves which metric, from PagedAttention to parallelism and hardware."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/llm-inference-optimization"
markdown_url: "https://theaiengineering.com/articles/llm-inference-optimization.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-10-23"
updated: "2026-10-23"
keywords: ["llm inference optimization","llm inference","llm inference engines","kv cache","llm quantization","speculative decoding","continuous batching","prefill vs decode"]
citation: "The AI Engineering Editors. \"LLM inference optimization: a practical guide for engineers.\" The AI Engineering, updated Oct 23, 2026. https://theaiengineering.com/articles/llm-inference-optimization"
---

# LLM inference optimization: a practical guide for engineers

## TL;DR

- LLM inference runs in two phases: prefill processes the whole prompt in parallel and is compute-bound, while decode generates one token per forward pass and is limited by memory bandwidth.
- The KV cache stores attention keys and values for every token in flight, and its size, together with the model weights, sets how many concurrent requests a GPU can serve.
- PagedAttention, continuous batching, and prefix caching raise throughput by wasting less GPU memory and fewer GPU cycles, and they do not change what the model generates.
- Quantization can change output quality and needs a task-specific eval, while speculative decoding with standard rejection sampling preserves the target model's output distribution.
- Measure time to first token, inter-token latency, and throughput under realistic traffic before and after each change, because most techniques improve one metric at the expense of another.

LLM inference optimization is the set of serving techniques that reduce the latency, memory footprint, and cost of generating tokens from a large language model while keeping output quality close to an unoptimized baseline. Most production gains come from a short list of methods: managing the KV cache efficiently, batching at the level of individual decode steps, quantizing weights, decoding speculatively, reusing shared prefixes, and splitting models across GPUs. This guide explains why each method works, which metric each one improves, and what to try first. It is part of the [LLM inference topic hub](https://theaiengineering.com/topics/llm-inference).

## What is LLM inference?

LLM inference is running a trained model on a prompt to generate output tokens one at a time, each conditioned on the prompt and all earlier tokens. Training happens once, but inference runs for every request a product serves, so inference efficiency sets both the serving bill and the latency users feel.

A decoder-only transformer produces one token per forward pass. Every request moves through [two phases](https://theaiengineering.com/glossary/prefill-and-decode) with different hardware bottlenecks, and most optimizations target one phase or the boundary between them.

### Prefill vs decode

- **Prefill** processes all prompt tokens in a single parallel forward pass, computes their attention keys and values, and emits the first output token. The work is dominated by large matrix multiplications, so prefill is compute-bound. Prefill time grows with prompt length and dominates time to first token.
- **Decode** generates one token per forward pass. Each step reads the full set of model weights and the request's cached keys and values from GPU memory but performs little arithmetic per byte read, so decode is limited by memory bandwidth at typical batch sizes. Decode step time sets [inter-token latency](https://theaiengineering.com/glossary/inter-token-latency).

**Figure 1: Prefill processes the prompt in parallel; decode emits one token per forward pass.**

One request with a long prompt and 200 output tokens: a single compute-bound prefill pass sets time to first token, then decode passes D1 to D200 each emit one token.

Components:

- **Prefill**: all prompt tokens in one parallel pass compute-bound
- **D1**: one token memory-bound
- **D2**: one token memory-bound
- **D200**: one token memory-bound

Groups:

- Time to first token: Prefill
- Inter-token latency × N: D1, D2, D200

Connections:

- Prefill → D1
- D1 → D2
- D2 → D200: ···

## Which metrics matter for LLM inference?

Four numbers describe most serving behavior: time to first token, inter-token latency, end-to-end latency, and throughput. Latency metrics describe one user's experience, while throughput describes the capacity of the server. Larger batches usually raise throughput and also raise per-request latency, so judge every optimization against an explicit latency target.

| Metric | What it measures | Dominated by | Watch out for |
|---|---|---|---|
| [Time to first token](https://theaiengineering.com/glossary/time-to-first-token) (TTFT) | Time from request arrival to the first output token | Queueing plus prefill of the prompt | Long prompts and prefill contention under load |
| Inter-token latency (ITL), also time per output token (TPOT) | Average gap between output tokens after the first | Decode step time at the current batch size | Large prefills scheduled alongside decodes |
| End-to-end latency | Total time to finish a response | Roughly TTFT plus ITL times the number of output tokens | Long outputs, including hidden reasoning tokens |
| Output throughput | Generated tokens per second across all requests | Batch size, memory bandwidth, KV cache capacity | Peak numbers measured without a latency limit |
| Request throughput | Completed requests per second | The mix of prompt and output lengths | Comparisons across different workloads |
| Goodput | Highest request rate that still meets latency targets | All of the above | The metric that maps most directly to cost per query |
| KV cache usage and preemptions | Share of cache blocks in use; requests evicted and recomputed | Concurrency and context length | Preemptions that inflate end-to-end latency |

Report latency as percentiles (p50, p95, p99) at a fixed request rate rather than as a single average. The [DistServe paper on disaggregated serving](https://arxiv.org/abs/2401.09670) uses goodput as its central metric, defined as the maximum request rate a deployment can serve while meeting its latency objectives. In production, the same metrics belong in your [LLM observability](https://theaiengineering.com/articles/llm-observability) dashboards.

## How does the KV cache work?

The [KV cache](https://theaiengineering.com/glossary/kv-cache) stores the attention key and value vectors computed for every previous token, so each decode step only computes vectors for the newest token instead of reprocessing the whole sequence. The cache grows linearly with sequence length and with the number of concurrent sequences, and it competes with model weights for GPU memory.

The per-token size follows directly from the model configuration:

```text
KV bytes per token = 2 (K and V) × layers × KV heads × head dim × bytes
```

Llama 3 70B has 80 layers, 8 key-value heads, and a head dimension of 128. At 16-bit precision (2 bytes per value), that is 327,680 bytes, or 320 KiB per token. One 8,192-token sequence needs 2.5 GiB of cache, and 32 such sequences need 80 GiB, on top of roughly 140 GB of 16-bit weights. The vLLM authors estimate 800 KB per token for OPT-13B, a smaller model that uses full multi-head attention without key-value head sharing.

Three levers shrink the cache:

- **Grouped-query attention (GQA)** shares each key-value head across several query heads. Llama 3 70B uses 8 key-value heads for 64 query heads, so its cache is one eighth the size full multi-head attention would need.
- **KV cache quantization** stores keys and values in 8 bits or fewer. vLLM, llama.cpp, and Ollama all document quantized KV cache options.
- **Context limits** cap the maximum sequence length per request, which bounds the worst-case cache per sequence.

### PagedAttention and memory fragmentation

Early serving systems reserved a contiguous memory region per request, sized for the maximum possible output. The [vLLM paper on PagedAttention](https://arxiv.org/abs/2309.06180) (Kwon et al., SOSP 2023) measured that only 20.4% to 38.2% of KV cache memory held actual token states in the existing systems it studied; reservation and fragmentation consumed the rest.

[PagedAttention](https://theaiengineering.com/glossary/paged-attention) applies operating-system paging to the cache. The engine splits each sequence's cache into fixed-size blocks, allocates blocks on demand, stores them anywhere in GPU memory, and lets requests share identical blocks. The authors report 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency. Block-based KV management is now standard: TensorRT-LLM documents a paged KV cache, and SGLang's RadixAttention is designed to work alongside paged attention.

## What is continuous batching?

[Continuous batching](https://theaiengineering.com/glossary/continuous-batching) schedules work one decode iteration at a time rather than one request at a time, so a finished sequence leaves the batch immediately and a waiting request joins at the next step. Output lengths vary widely between requests, and iteration-level scheduling keeps the GPU busy instead of idling until the longest sequence in a static batch finishes.

The idea was introduced as iteration-level scheduling in [Orca, presented at OSDI 2022](https://www.usenix.org/conference/osdi22/presentation/yu), which paired it with selective batching of operations that tolerate sequences of different lengths. vLLM, SGLang, TensorRT-LLM (where the feature is called in-flight batching), and llama.cpp's server all implement continuous batching today. Continuous batching also explains why per-request benchmarks mislead: a single request on an idle server measures latency, not the throughput the same server delivers at 50 concurrent users.

### Chunked prefill

A long prompt arriving mid-stream can stall every in-flight decode while its prefill runs. Chunked prefill splits large prefills into pieces and mixes them with decode steps. In vLLM's scheduler, pending decodes are batched first, and the remaining per-step token budget (`max_num_batched_tokens`) is filled with prefill chunks. vLLM's documentation describes the trade-off: a smaller budget improves inter-token latency, and a larger budget improves time to first token and throughput.

## LLM quantization: what to quantize and what it costs

[Quantization](https://theaiengineering.com/glossary/quantization) stores weights, activations, or the KV cache in fewer bits than the 16-bit format most models ship in. Fewer bits mean less memory for the model and fewer bytes to read per decode step, which raises the batch a GPU can hold and speeds up memory-bound decoding. Lower precision can also reduce accuracy, so evaluate every quantized model.

| Scheme | What is quantized | Example support | Hardware note |
|---|---|---|---|
| INT4 weight-only (GPTQ, AWQ) | Weights only; activations stay 16-bit | vLLM | Gains are largest when decode is memory-bound, typically at small batch sizes |
| FP8 (W8A8) | Weights and activations | vLLM, TensorRT-LLM | vLLM documents official W8A8 support on Ada Lovelace and Hopper GPUs (compute capability 8.9 and above) |
| FP4 (NVFP4, MXFP4) | Weights, often activations | vLLM, TensorRT-LLM | NVIDIA introduced NVFP4 with the Blackwell architecture |
| GGUF k-quants (for example Q4_K_M) | Weights, in 1.5-bit to 8-bit integer formats | llama.cpp, Ollama, LM Studio | Runs on CPUs, Apple silicon, and consumer GPUs |
| 8-bit KV cache | Cached keys and values | vLLM, llama.cpp, Ollama | Doubles cache capacity relative to 16-bit; check long-context quality |

The GPTQ paper (Frantar et al., ICLR 2023) reports quantizing GPT-family models to 3 or 4 bits per weight with little accuracy loss on its benchmarks. Treat published results as a starting point, not a guarantee: re-run your task-specific suite from your [LLM evaluation](https://theaiengineering.com/articles/llm-evaluation) process after quantizing, because regressions often show up in formatting, rare languages, or long-context retrieval before they show up in aggregate scores.

## What is speculative decoding?

[Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding) uses a cheap draft mechanism to propose several tokens, then runs the target model once to verify all of them and keeps the longest accepted run. Because decode steps are memory-bound, verifying several tokens costs about the same as generating one, so each accepted draft token reduces inter-token latency.

[Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) (Leviathan et al., ICML 2023) showed that a rejection-sampling verification step makes the output distribution identical to the target model's, and reported 2 to 3 times faster decoding on T5-XXL. Current engines offer several draft sources:

- **Draft models**: a smaller model from the same family with a shared vocabulary.
- **EAGLE and multi-token prediction (MTP) heads**: light prediction layers attached to the target model.
- **N-gram and suffix matching**: proposals copied from text already in the prompt or earlier output, which suit code edits and extraction tasks that repeat input text.

vLLM's documentation targets speculative decoding at medium-to-low request rates, where decode is memory-bound. Under heavy load the GPU becomes compute-bound, rejected drafts waste capacity, and the speedup shrinks. Measure the acceptance rate on your own traffic before enabling it everywhere.

## What is prefix caching?

Prefix caching keeps the KV cache blocks of completed requests and reuses them when a new request begins with the same tokens, which skips prefill for the shared part. Long system prompts, few-shot examples, multi-turn chat history, and agent tool definitions all benefit, because every turn resends the same prefix.

vLLM's automatic prefix caching hashes each full cache block together with its parent block's hash, so a block matches only when the entire preceding prefix matches. vLLM can mix a per-request cache salt into the hash to keep tenants from sharing cache entries. SGLang stores prefixes in a radix tree (RadixAttention) with least-recently-used eviction, and llama.cpp's server reuses the KV cache from a previous request through its `cache_prompt` option.

Hosted APIs offer a comparable feature called [prompt caching](https://theaiengineering.com/glossary/prompt-caching), usually with discounted pricing for cached input tokens. Prefix caching is different from semantic caching, which returns a stored response for a similar query and usually runs at an [AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway).

To raise hit rates, place stable content (instructions, tools, reference documents) first and variable content last, a habit covered in [context engineering](https://theaiengineering.com/articles/context-engineering).

## Tensor parallelism vs pipeline parallelism

[Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism) splits each layer's weight matrices across GPUs, so every GPU works on every token and the GPUs exchange activations inside each layer. Pipeline parallelism assigns consecutive groups of layers to different GPUs or nodes, which needs less communication but leaves stages idle while the pipeline fills. Use tensor parallelism inside a node and add pipeline parallelism across nodes.

vLLM's scaling guidance maps cleanly to deployment size:

- **Model fits on one GPU**: no model parallelism; add replicas (data parallelism) for throughput.
- **Model fits on one node**: set `--tensor-parallel-size` to the number of GPUs in the node.
- **Model needs several nodes**: tensor parallelism within each node, and `--pipeline-parallel-size` equal to the number of nodes.
- **GPUs without NVLink**: vLLM recommends pipeline parallelism, because tensor parallelism's per-layer communication saturates slower links.

Mixture-of-experts models add expert parallelism, which places different experts on different GPUs and routes tokens between them.

### Prefill-decode disaggregation

Prefill and decode compete when they share GPUs: a burst of long prompts slows every in-flight decode. Disaggregated serving runs the two phases on separate GPU pools and transfers the KV cache between them. DistServe reported serving up to 7.4 times more requests, or meeting 12.6 times tighter latency targets, than colocated baselines. vLLM, SGLang, and TensorRT-LLM (in beta) now support disaggregated prefill and decode; the cost is a fast network for KV transfer and a more complex deployment.

## How to choose a GPU for LLM inference

Choose hardware by four properties, in order: memory capacity for weights plus KV cache at your target concurrency, memory bandwidth for decode speed, supported low-precision formats such as FP8 and FP4, and GPU-to-GPU interconnect if the model needs tensor parallelism. Peak compute matters most for prefill-heavy work such as long-document processing.

A sizing pass takes three steps:

1. **Weights**: parameters times bytes per parameter. A 70B model needs about 140 GB at 16-bit, about 70 GB at FP8, and about 35 GB at 4-bit, plus quantization scales and runtime overhead.
2. **KV cache**: per-token size (from the formula above) times context length times concurrent sequences.
3. **Decode ceiling**: at batch size 1, each token requires reading all weights once, so memory bandwidth divided by weight bytes approximates the upper bound on single-stream tokens per second. Batching raises total throughput by sharing that read across sequences.

Apple silicon Macs and consumer GPUs offer less memory bandwidth than data center GPUs at a lower cost, and Apple's unified memory architecture shares one memory pool between CPU and GPU, which suits local development; the [vLLM vs Ollama comparison](https://theaiengineering.com/articles/vllm-vs-ollama) covers engines built for those machines. If operating GPUs is not a requirement, hosted [inference providers](https://theaiengineering.com/platforms/inference-providers) sell the same optimizations as a per-token service.

## Which optimization should you try first?

Start with changes that modern engines provide at no quality cost, then move to changes that alter the model or the deployment topology. A practical order: adopt an engine with paged KV and continuous batching, enable prefix caching, tune batch limits to your latency target, quantize, add speculative decoding, and only then scale out with parallelism or disaggregation.

| Technique | Primarily improves | Main cost or risk | Changes outputs? |
|---|---|---|---|
| Paged KV cache | Concurrency, throughput | None in engines that ship it | No |
| Continuous batching | Throughput, queueing delay | Higher ITL at very large batches | No |
| Chunked prefill | ITL stability under load | Tuning the per-step token budget | No |
| Prefix caching | TTFT, prefill compute | Memory held by cached blocks; tenant isolation | No |
| Weight quantization | Memory, decode speed, cost | Possible quality loss; kernel support varies | Can |
| KV cache quantization | Concurrency, context length | Possible long-context quality loss | Can |
| Speculative decoding | ITL at low-to-medium load | Draft overhead; gains shrink at high load | No, with standard rejection sampling |
| Tensor or pipeline parallelism | Fits larger models; per-token latency | Interconnect bandwidth, idle pipeline stages | No |
| Prefill-decode disaggregation | Goodput under strict TTFT and ITL targets | KV transfer network, operational complexity | No |

"No" in the last column means the technique preserves the model's output distribution; results can still differ at the floating-point level across batch sizes and kernels.

To pick an engine that implements these techniques, read the [vLLM vs Ollama comparison](https://theaiengineering.com/articles/vllm-vs-ollama) and browse the [serving engines directory](https://theaiengineering.com/platforms/serving-engines). If your traffic includes long chains of hidden reasoning, [reasoning models](https://theaiengineering.com/articles/reasoning-models) explains why those workloads are decode-heavy and how effort settings change their cost.

## Frequently asked questions

### What is prefill and decode in LLM inference?

Prefill is the first phase of a request: the model processes every prompt token in one parallel forward pass, fills the KV cache, and emits the first output token. Decode is the second phase: the model generates one token per forward pass, reading all weights and the cached keys and values each step. Prefill is compute-bound and drives time to first token; decode is memory-bound and drives inter-token latency.

### How do you speed up LLM inference?

Start with a serving engine that implements paged KV cache management and continuous batching, such as vLLM, SGLang, or TensorRT-LLM. Then enable prefix caching for repeated prompt prefixes, tune batch token limits for your latency target, quantize weights to FP8 or 4-bit formats after checking quality, and add speculative decoding for latency at low to medium load. Use tensor parallelism when the model does not fit on one GPU.

### What does the KV cache store?

The KV cache stores the key and value vectors that each attention layer computed for every previous token in a sequence. Reusing those vectors means each decode step only computes keys and values for the newest token. Cache size per token equals two times the number of layers, key-value heads, head dimension, and bytes per value, so long contexts and many concurrent requests consume GPU memory quickly.

### What is a good time to first token?

No universal number applies, because acceptable time to first token depends on the product. Streaming chat interfaces need the first token fast enough that the interface feels responsive, while batch summarization jobs can tolerate seconds. Set a target per route from user research or product requirements, measure p50 and p95 under realistic load, and remember that long prompts, queueing, and hidden reasoning tokens all delay the first visible token.

### Does LLM quantization reduce quality?

LLM quantization can reduce quality, and the size of the loss depends on the format, the method, the model, and the task. Published methods such as GPTQ report small accuracy losses at 3 or 4 bits per weight on the benchmarks they test, and 8-bit formats discard less information than 4-bit formats. Run your own evaluation set on the quantized model before shipping, because aggregate benchmark scores can hide task-specific regressions.

### What is the best GPU for LLM inference?

The best GPU for LLM inference is the one whose memory holds the model weights plus the KV cache for your target concurrency, with enough memory bandwidth for your decode latency target. After that, check support for the precision formats you plan to use, such as FP8 or FP4, and the interconnect between GPUs if the model needs tensor parallelism. Benchmark your actual model and traffic before buying or reserving capacity.

## Terms in this guide

- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Mixture of experts](https://theaiengineering.com/glossary/mixture-of-experts): Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token.
- [Tokens](https://theaiengineering.com/glossary/tokens): Tokens are the units of text a language model processes, typically whole words, word fragments, punctuation, or bytes, each mapped to an integer ID from the model's fixed vocabulary.
- [Large language model (LLM)](https://theaiengineering.com/glossary/large-language-model): A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on web-scale text to predict the next token and then adapted to follow instructions.
- [Transformer](https://theaiengineering.com/glossary/transformer): A transformer is a neural network architecture that processes sequences with self-attention, letting each token weigh information from other tokens in parallel; it is the basis of most modern large language models.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.
- [Few-shot prompting](https://theaiengineering.com/glossary/few-shot-prompting): Few-shot prompting is a technique that puts a few example inputs paired with desired outputs in a prompt, so a language model infers the task, format, and labels without changing its weights.
- [Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism): Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.
- [Test-time compute](https://theaiengineering.com/glossary/test-time-compute): Test-time compute is the computation a language model spends while answering a query, through longer reasoning, multiple samples, or search, as opposed to the train-time compute spent building the model.

## Related guides

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this guide

The AI Engineering Editors. "LLM inference optimization: a practical guide for engineers." The AI Engineering, updated Oct 23, 2026. https://theaiengineering.com/articles/llm-inference-optimization

- Title: LLM inference optimization: a practical guide for engineers
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-10-23
- Updated: 2026-10-23
- URL: https://theaiengineering.com/articles/llm-inference-optimization
- Markdown: https://theaiengineering.com/articles/llm-inference-optimization.md

To cite one section, link to its anchor:

- [What is LLM inference?](https://theaiengineering.com/articles/llm-inference-optimization#what-is-llm-inference)
- [Which metrics matter for LLM inference?](https://theaiengineering.com/articles/llm-inference-optimization#which-metrics-matter-for-llm-inference)
- [How does the KV cache work?](https://theaiengineering.com/articles/llm-inference-optimization#how-does-the-kv-cache-work)
- [What is continuous batching?](https://theaiengineering.com/articles/llm-inference-optimization#what-is-continuous-batching)
- [LLM quantization: what to quantize and what it costs](https://theaiengineering.com/articles/llm-inference-optimization#llm-quantization-what-to-quantize-and-what-it-costs)
- [What is speculative decoding?](https://theaiengineering.com/articles/llm-inference-optimization#what-is-speculative-decoding)
- [What is prefix caching?](https://theaiengineering.com/articles/llm-inference-optimization#what-is-prefix-caching)
- [Tensor parallelism vs pipeline parallelism](https://theaiengineering.com/articles/llm-inference-optimization#tensor-parallelism-vs-pipeline-parallelism)
- [How to choose a GPU for LLM inference](https://theaiengineering.com/articles/llm-inference-optimization#how-to-choose-a-gpu-for-llm-inference)
- [Which optimization should you try first?](https://theaiengineering.com/articles/llm-inference-optimization#which-optimization-should-you-try-first)
- [Frequently asked questions](https://theaiengineering.com/articles/llm-inference-optimization#faq)
