What is an LLM inference engine? Examples and uses
Also called LLM serving engine, inference server, model server, LLM inference engine
Definition
An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
An inference engine is the serving software that sits between a model's weights and the requests an application sends. It loads the weights onto GPUs or CPUs, groups concurrent requests into batches, allocates KV cache memory, runs the forward passes, and streams tokens back, usually over an OpenAI-compatible API. The engine decides how many users one machine can serve and how quickly each response starts. Open-source examples include vLLM, SGLang, TensorRT-LLM, llama.cpp, and Ollama.
What an inference engine does#
An inference engine turns a model checkpoint into a service. Its core jobs are loading weights in a supported format and precision, scheduling requests, managing KV cache memory, and exposing an HTTP API. Production engines add throughput techniques that keep the GPU busy across many concurrent requests without changing what the model generates.
vLLM's documentation lists a typical feature set for a GPU serving engine: PagedAttention for KV cache memory, continuous batching, chunked prefill, prefix caching, and speculative decoding. It also supports quantization formats from FP8 and INT4 to GGUF, and tensor, pipeline, data, expert, and context parallelism, behind an OpenAI-compatible server that also offers the Anthropic Messages API and gRPC.
Engines differ mainly in hardware target, model format, and whether they serve one user or many:
| Engine | Maintainer | Built for | Model format |
|---|---|---|---|
| vLLM | vLLM project | Multi-user serving on GPUs and other accelerators | Hugging Face checkpoints |
| SGLang | Hosted by LMSYS | Serving language, vision-language, and diffusion models | Hugging Face checkpoints |
| TensorRT-LLM | NVIDIA | Serving on NVIDIA GPUs | Hugging Face checkpoints |
| llama.cpp | ggml-org | Portable inference on CPUs and many GPU backends | GGUF |
| Ollama | Ollama | Running models on one machine through a CLI, app, and API | GGUF and safetensors imports |
| Hugging Face TGI | Hugging Face | GPU serving; now in maintenance mode | Hugging Face checkpoints |
Hugging Face archived the Text Generation Inference repository on March 21, 2026. Its notice says the project is in maintenance mode and points users to vLLM and SGLang for GPU serving and to llama.cpp or MLX for local use.
Inference engine vs inference provider#
An inference engine is software you run on your own hardware, while an inference provider is a company that runs models on its hardware and sells access through an API, usually priced per token or per hour of reserved capacity. Running an engine gives control over models, data, and tuning. Using a provider removes GPU operations.
| Inference engine | Inference provider | |
|---|---|---|
| What it is | Software, often open source | A hosted service |
| Who runs the GPUs | Your team | The provider |
| How you pay | For your own hardware or cloud instances | Per token or for reserved capacity |
| What you tune | Batch limits, quantization, parallelism, caching | Model choice and request parameters |
The two meet at the API. Most engines and many providers accept OpenAI-style requests, so an application can move between a self-hosted engine and a hosted provider by changing the base URL and model name. Behavior still differs in tool calling formats, structured output enforcement, and reasoning-token handling, so test those paths when switching. Compare hosted options in the inference providers directory.
vLLM vs Ollama compares six engines feature by feature, and the serving engines directory lists more.
Related terms
Guides that cover inference engine
Cite this definition
The AI Engineering Editors. "What is an LLM inference engine? Examples and uses." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inference-engine
- Title
- What is an LLM inference engine? Examples and uses
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated