Topic hub 07

LLM inference and serving

Serving models fast and cheaply: engines, memory, batching and hardware.

Overview

LLM inference is the process of running a trained model to generate output, and it is where most of an AI product's latency and compute cost comes from. Serving engines manage GPU memory, batch requests and stream tokens so a model can serve many users at once.

This hub covers the mechanics of prefill and decode, the KV cache, quantization and batching, and how the main serving engines and inference providers compare.

LLM inference and serving guides

Key terms

Full glossary

Tools and platforms

Platform directory

Other topics