---
title: "LLM inference and serving"
description: "LLM inference explained for engineers: serving engines such as vLLM and Ollama, KV cache, batching, quantization and the metrics that decide cost and latency."
type: "topic-hub"
canonical_url: "https://theaiengineering.com/topics/llm-inference"
markdown_url: "https://theaiengineering.com/topics/llm-inference.md"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
keywords: ["llm inference","llm inference optimization","vllm vs ollama","kv cache","llm quantization","llm inference engines"]
citation: "The AI Engineering Editors. \"LLM inference and serving.\" The AI Engineering. https://theaiengineering.com/topics/llm-inference"
---

# LLM inference and serving

Serving models fast and cheaply: engines, memory, batching and hardware.

LLM inference is the process of running a trained model to generate output, and it is where most of an AI product's latency and compute cost comes from. Serving engines manage GPU memory, batch requests and stream tokens so a model can serve many users at once.

This hub covers the mechanics of prefill and decode, the KV cache, quantization and batching, and how the main serving engines and inference providers compare.

## LLM inference and serving guides

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.

## Key terms

- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [Inter-token latency](https://theaiengineering.com/glossary/inter-token-latency): Inter-token latency (ITL) is the average time between consecutive output tokens of an LLM response after the first token, which sets how fast streamed text appears once generation starts.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [LLM streaming](https://theaiengineering.com/glossary/llm-streaming): LLM streaming is the delivery of a language model's output in small chunks as it is generated, usually as server-sent events over HTTP, so clients can show text before the response is complete.
- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [Prefill and decode](https://theaiengineering.com/glossary/prefill-and-decode): Prefill and decode are the two phases of LLM inference: prefill processes the whole prompt in one parallel pass and fills the KV cache, and decode generates one output token per forward pass.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [Speculative decoding](https://theaiengineering.com/glossary/speculative-decoding): Speculative decoding is an inference technique where a cheap draft proposes a few tokens and the target model verifies them in one forward pass, emitting several tokens per step without changing the output distribution.
- [Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism): Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.

## Tools and platforms

- [Serving engines](https://theaiengineering.com/platforms/serving-engines): 34 platforms. An LLM serving engine is software that loads model weights onto GPUs or CPUs and generates tokens in response to requests, managing batching, KV cache memory, and an HTTP API.
- [Inference providers](https://theaiengineering.com/platforms/inference-providers): 28 platforms. An inference provider is a service that runs trained models on its own hardware and sells access to them through an API, so teams can use open-weight or proprietary models without operating GPUs.
- [GPU clouds](https://theaiengineering.com/platforms/gpu-clouds): 33 platforms. A GPU cloud is a provider that rents GPUs and other AI accelerators by the hour, the month, or under multi-year contracts, so teams can train, fine-tune, and serve models without buying hardware.

## Cite this page

The AI Engineering Editors. "LLM inference and serving." The AI Engineering. https://theaiengineering.com/topics/llm-inference

- Title: LLM inference and serving
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/topics/llm-inference
- Markdown: https://theaiengineering.com/topics/llm-inference.md
