---
title: "What is an LLM inference engine? Examples and uses"
description: "An inference engine loads LLM weights, batches requests, manages the KV cache, and serves tokens over an API. Examples, and how engines differ from providers."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/inference-engine"
markdown_url: "https://theaiengineering.com/glossary/inference-engine.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Inference engine","LLM serving engine","inference server","model server","LLM inference engine"]
citation: "The AI Engineering Editors. \"What is an LLM inference engine? Examples and uses.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inference-engine"
---

# What is an LLM inference engine? Examples and uses

Also called LLM serving engine, inference server, model server, LLM inference engine.

> **Definition of Inference engine:** An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.

An inference engine is the serving software that sits between a model's weights and the requests an application sends. It loads the weights onto GPUs or CPUs, groups concurrent requests into batches, allocates [KV cache](https://theaiengineering.com/glossary/kv-cache) memory, runs the forward passes, and streams tokens back, usually over an [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api). The engine decides how many users one machine can serve and how quickly each response starts. Open-source examples include vLLM, SGLang, TensorRT-LLM, llama.cpp, and Ollama.

## What an inference engine does

An inference engine turns a model checkpoint into a service. Its core jobs are loading weights in a supported format and precision, scheduling requests, managing KV cache memory, and exposing an HTTP API. Production engines add throughput techniques that keep the GPU busy across many concurrent requests without changing what the model generates.

[vLLM's documentation](https://docs.vllm.ai/en/latest/) lists a typical feature set for a GPU serving engine: [PagedAttention](https://theaiengineering.com/glossary/paged-attention) for KV cache memory, [continuous batching](https://theaiengineering.com/glossary/continuous-batching), chunked prefill, prefix caching, and [speculative decoding](https://theaiengineering.com/glossary/speculative-decoding). It also supports quantization formats from FP8 and INT4 to GGUF, and tensor, pipeline, data, expert, and context parallelism, behind an OpenAI-compatible server that also offers the Anthropic Messages API and gRPC.

Engines differ mainly in hardware target, model format, and whether they serve one user or many:

| Engine | Maintainer | Built for | Model format |
|---|---|---|---|
| vLLM | vLLM project | Multi-user serving on GPUs and other accelerators | Hugging Face checkpoints |
| SGLang | Hosted by LMSYS | Serving language, vision-language, and diffusion models | Hugging Face checkpoints |
| TensorRT-LLM | NVIDIA | Serving on NVIDIA GPUs | Hugging Face checkpoints |
| llama.cpp | ggml-org | Portable inference on CPUs and many GPU backends | [GGUF](https://theaiengineering.com/glossary/gguf) |
| Ollama | Ollama | Running models on one machine through a CLI, app, and API | GGUF and safetensors imports |
| Hugging Face TGI | Hugging Face | GPU serving; now in maintenance mode | Hugging Face checkpoints |

Hugging Face archived the [Text Generation Inference repository](https://github.com/huggingface/text-generation-inference) on March 21, 2026. Its notice says the project is in maintenance mode and points users to vLLM and SGLang for GPU serving and to llama.cpp or MLX for local use.

## Inference engine vs inference provider

An inference engine is software you run on your own hardware, while an inference provider is a company that runs models on its hardware and sells access through an API, usually priced per token or per hour of reserved capacity. Running an engine gives control over models, data, and tuning. Using a provider removes GPU operations.

| | Inference engine | Inference provider |
|---|---|---|
| What it is | Software, often open source | A hosted service |
| Who runs the GPUs | Your team | The provider |
| How you pay | For your own hardware or cloud instances | Per token or for reserved capacity |
| What you tune | Batch limits, quantization, parallelism, caching | Model choice and request parameters |

The two meet at the API. Most engines and many providers accept OpenAI-style requests, so an application can move between a self-hosted engine and a hosted provider by changing the base URL and model name. Behavior still differs in tool calling formats, structured output enforcement, and reasoning-token handling, so test those paths when switching. Compare hosted options in the [inference providers directory](https://theaiengineering.com/platforms/inference-providers).

[vLLM vs Ollama](https://theaiengineering.com/articles/vllm-vs-ollama) compares six engines feature by feature, and the [serving engines directory](https://theaiengineering.com/platforms/serving-engines) lists more.

## Related terms

- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.

## Guides that cover inference engine

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [RAG vs fine-tuning: how to choose for your LLM application](https://theaiengineering.com/articles/rag-vs-fine-tuning): RAG vs fine-tuning: RAG adds knowledge at query time, while fine-tuning changes model behavior. Compare freshness, cost, latency, data needs, LoRA, and hybrids.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this definition

The AI Engineering Editors. "What is an LLM inference engine? Examples and uses." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/inference-engine

- Title: What is an LLM inference engine? Examples and uses
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/inference-engine
- Markdown: https://theaiengineering.com/glossary/inference-engine.md
