Serving engines
An LLM serving engine is software that loads model weights onto GPUs or CPUs and generates tokens in response to requests, managing batching, KV cache memory, and an HTTP API. The engine determines how many concurrent users one machine can serve and how quickly each response starts.
Production engines such as vLLM, SGLang, TensorRT-LLM, and TokenSpeed target multi-user GPU serving with continuous batching, prefix caching, quantization, and multi-GPU parallelism, while NVIDIA Dynamo, llm-d, and KServe coordinate those engines across clusters. Local tools such as llama.cpp, Ollama, LM Studio, and MLX-LM focus on running quantized models on a single laptop or workstation.
Most engines expose an OpenAI-compatible endpoint, so applications can move between them with few code changes. The choice depends mostly on hardware, model format, and whether one person or many users share the deployment. Hugging Face's TGI is now in maintenance mode, so new deployments usually start with another engine.
Serving engines compared
| Platform | Open source | Deployment | Best for |
|---|---|---|---|
| BentoML | Yes (Apache-2.0) | Self-hosted, Library | Packaging custom inference code as production APIs |
| Docker Model Runner | Yes (Apache-2.0) | Local, CLI, Desktop app | Teams that already build and ship with Docker |
| ExLlamaV3 | Yes (MIT) | Local, Library | Large quantized models on consumer NVIDIA GPUs |
| exo | Yes (Apache-2.0) | Local, Desktop app | Running large models across several home devices |
| Foundry Local | No | Local, Library, CLI | Shipping local inference inside Windows and cross-platform apps |
| Jan | Yes (Apache-2.0) | Desktop app, Local | An open-source desktop chat app for local models |
| KoboldCpp | Yes (AGPL-3.0) | Local, Desktop app | Zero-install local models for chat and writing |
| KServe | Yes (Apache-2.0) | Self-hosted | Standardized model serving on Kubernetes |
| KTransformers | Yes (Apache-2.0) | Self-hosted, Library | Large MoE models on a single workstation |
| Lemonade | Yes (Apache-2.0) | Local, Desktop app | Local models on AMD Ryzen AI and Radeon PCs |
| LightLLM | Yes (Apache-2.0) | Self-hosted, Library | A compact Python codebase for serving research |
| LiteRT-LM | Yes (Apache-2.0) | Local, Library, CLI | On-device Gemma and small models in mobile and web apps |
| llama.cpp | Yes (MIT) | Local, Self-hosted, Library, CLI | Quantized models on laptops, desktops, and edge devices |
| llamafile | Yes (Apache-2.0) | Local, CLI | Distributing a model as one portable executable |
| llm-d | Yes (Apache-2.0) | Self-hosted | Running vLLM at scale on Kubernetes |
| LM Studio | No | Desktop app, Local, CLI | A graphical app for local models on one machine |
| LMDeploy | Yes (Apache-2.0) | Self-hosted, Library | Quantized LLM serving on NVIDIA GPUs |
| LocalAI | Yes (MIT) | Local, Self-hosted | A self-hosted drop-in for several OpenAI APIs |
| mistral.rs | Yes (MIT) | Local, Self-hosted, Library | Rust applications that embed local inference |
| MLC LLM | Yes (Apache-2.0) | Local, Self-hosted, Library | One model build for phones, browsers, and GPUs |
| MLX-LM | Yes (MIT) | Local, Library, CLI | Running and fine-tuning models on Apple Silicon Macs |
| Modular MAX | No | Self-hosted, Library | Portable serving across NVIDIA, AMD, and Apple hardware |
| NVIDIA Dynamo | Yes (Apache-2.0) | Self-hosted | Multi-node serving across large GPU clusters |
| NVIDIA Dynamo-Triton (formerly Triton Inference Server) | Yes (BSD-3-Clause) | Self-hosted | Serving mixed model types from one server |
| Ollama | Yes (MIT) | Local, Self-hosted, CLI, Desktop app, Managed cloud | Running open models locally with minimal setup |
| RamaLama | Yes (MIT) | Local, CLI | Running local models in containers with Podman or Docker |
| SGLang | Yes (Apache-2.0) | Self-hosted, Library | Workloads with shared prefixes and large MoE models |
| Sonar (formerly Aphrodite Engine) | Yes (AGPL-3.0) | Self-hosted, Library | vLLM-style serving with extra samplers and quant formats |
| TensorRT-LLM | Yes (Apache-2.0) | Self-hosted, Library | Maximizing performance on NVIDIA GPU fleets |
| Text Generation Inference (TGI) | Yes (Apache-2.0) | Self-hosted | Maintaining existing TGI deployments |
| TextGen (formerly text-generation-webui) | Yes (AGPL-3.0) | Desktop app, Local | Experimenting with local models across several backends |
| TokenSpeed | Yes (MIT) | Self-hosted, Library | Agentic workloads on recent NVIDIA and AMD GPUs |
| vLLM | Yes (Apache-2.0) | Self-hosted, Library | High-throughput multi-user serving on GPUs |
| Xinference | Yes (Apache-2.0) | Self-hosted, Local | One API over several engines and model types |
Profiles
BentoML
Modular · bentoml.com
BentoML is a Python framework for packaging models and inference code into APIs, job queues, and multi-model pipelines, with integrations for vLLM and other engines and container images for deployment anywhere.
Modular acquired BentoML in February 2026, and the BentoML framework remains open source under Apache-2.0.

Docker Model Runner
Docker · docs.docker.com
Docker Model Runner pulls models as OCI artifacts from Docker Hub or other registries and runs them locally through Docker Desktop or Docker Engine with llama.cpp, vLLM, or SGLang behind an OpenAI-compatible API.

ExLlamaV3
github.com
ExLlamaV3 is an MIT-licensed quantization and inference library for running LLMs on consumer NVIDIA GPUs with its EXL3 format; it succeeds ExLlamaV2, whose last release came in July 2025.

exo
Exo Labs · exolabs.net
exo connects Macs and other devices on a local network into one cluster for running models too large for a single machine, splitting each model across devices and exposing OpenAI-, Claude-, and Ollama-compatible APIs.
Foundry Local
Microsoft · foundrylocal.ai
Foundry Local is Microsoft's on-device inference runtime that downloads optimized models and runs them with ONNX Runtime on CPUs, GPUs, and NPUs, shipping inside applications through SDKs with no Azure subscription required.
Jan
Menlo Research · jan.ai
Jan is an Apache-2.0 desktop app from Menlo Research that runs open models locally through llama.cpp, connects to cloud model APIs, and exposes a local OpenAI-compatible server for other applications.
KoboldCpp
github.com
KoboldCpp is a single-file program built on llama.cpp that runs GGUF models locally with the bundled KoboldAI Lite interface and serves KoboldAI, OpenAI-compatible, and Ollama-compatible APIs, plus image and speech generation.

KServe
kserve.github.io
KServe is a CNCF incubating project for serving predictive and generative models on Kubernetes, providing InferenceService resources, autoscaling, OpenAI-compatible endpoints, and vLLM and llm-d backends for LLMs.

KTransformers
github.com
KTransformers is a framework for heterogeneous CPU and GPU inference and fine-tuning that keeps parts of large mixture-of-experts models in system memory, so they can run on machines with limited GPU memory.

Lemonade
lemonade-server.ai
Lemonade is a local AI server that runs language, speech, and image models on PC GPUs and NPUs, with optimizations from AMD engineers for Ryzen AI and Radeon hardware and OpenAI-, Anthropic-, and Ollama-compatible APIs.

LightLLM
github.com
LightLLM is a Python-based LLM inference and serving framework from ModelTC with a lightweight design, token-level KV cache management, and multi-GPU support, aimed at research and production serving.

LiteRT-LM
Google · ai.google.dev
LiteRT-LM is Google's runtime for running LLMs on device with LiteRT across mobile, web, desktop, and Raspberry Pi, with GPU acceleration, constrained decoding, and speculative decoding for models such as Gemma.

llama.cpp
github.com
llama.cpp is an MIT-licensed C/C++ inference library and server that runs quantized models in the GGUF format on CPUs, Apple Silicon, and GPUs; its maintainers at ggml.ai joined Hugging Face in February 2026.

llamafile
Mozilla.ai · github.com
llamafile is a Mozilla.ai project that bundles model weights and a llama.cpp-based runtime into a single executable that runs on macOS, Windows, Linux, and BSD without installation, using Cosmopolitan Libc.

llm-d
llm-d.ai
llm-d is a Kubernetes-native distributed inference stack and CNCF sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, that adds cache-aware routing and disaggregated serving above vLLM or SGLang.
LM Studio
lmstudio.ai
LM Studio is a desktop app for macOS, Windows, and Linux that downloads and runs local models with llama.cpp or Apple MLX, serves them on OpenAI-compatible endpoints, and includes Bionic, an agent for documents and code.

LMDeploy
github.com
LMDeploy, from the InternLM team at Shanghai AI Laboratory, is a toolkit for compressing, deploying, and serving LLMs with its C++ TurboMind engine or a PyTorch engine, with quantization and an OpenAI-compatible server.

LocalAI
localai.io
LocalAI is an MIT-licensed, OpenAI-compatible server that runs language, vision, speech, image, and video models on local hardware with or without a GPU, switching between backends such as llama.cpp, vLLM, and diffusers.

mistral.rs
github.com
mistral.rs is an LLM inference engine written in Rust with in-situ quantization, multimodal model support, an OpenAI-compatible server, and Python and Rust APIs for embedding inference in applications.
MLC LLM
llm.mlc.ai
MLC LLM is a compiler-based engine that uses Apache TVM to compile models for GPUs, Apple Silicon, iOS, Android, and web browsers through WebLLM, exposing an OpenAI-compatible API on each platform.

MLX-LM
Apple · github.com
MLX-LM is an MIT-licensed Python package from Apple's MLX team for running and fine-tuning LLMs on Apple Silicon with the MLX array framework, including quantization, LoRA training, and an OpenAI-compatible HTTP server.
Modular MAX
Modular · modular.com
MAX is Modular's serving and modeling framework that runs the same package on NVIDIA, AMD, and Apple silicon, exposes OpenAI-compatible APIs, and uses kernels written in Mojo; its source is partly Apache-2.0 and partly under the MAX license.

NVIDIA Dynamo
NVIDIA · developer.nvidia.com
NVIDIA Dynamo is an open-source orchestration layer for datacenter-scale inference that coordinates vLLM, SGLang, or TensorRT-LLM across nodes, adding disaggregated prefill and decode, KV-aware routing, multi-tier KV caching, and autoscaling.

NVIDIA Dynamo-Triton (formerly Triton Inference Server)
NVIDIA · developer.nvidia.com
Dynamo-Triton is NVIDIA's open-source inference server for deploying models from TensorRT, TensorRT-LLM, PyTorch, ONNX, vLLM, and other backends, with dynamic batching, model ensembles, and HTTP and gRPC endpoints.
NVIDIA renamed Triton Inference Server to Dynamo-Triton, and the GitHub repository keeps the triton-inference-server name.
Ollama
ollama.com
Ollama is an MIT-licensed tool for downloading and running open models on macOS, Windows, and Linux through a CLI, a desktop app, and a local REST API, with an option to run larger models on Ollama's cloud.
RamaLama
ramalama.ai
RamaLama is a command-line tool from the containers community that runs AI models inside OCI containers, detecting the host's GPU and pulling a matching llama.cpp or vLLM image so models run in isolation.

SGLang
sglang.io
SGLang is an open-source serving framework for language and multimodal models that uses RadixAttention to reuse KV cache across requests with shared prefixes, and supports prefill-decode disaggregation and expert parallelism.
Sonar (formerly Aphrodite Engine)
sonar.dphn.ai
Sonar is an inference engine based on vLLM that adds extra quantization formats, sampling methods, and kernels, and serves production traffic for the Dolphin inference network and PygmalionAI.
Aphrodite Engine was renamed Sonar and moved to the dphnAI GitHub organization, and its command-line tool still uses the aphrodite name.

TensorRT-LLM
NVIDIA · developer.nvidia.com
TensorRT-LLM is NVIDIA's open-source library for optimizing LLM inference on NVIDIA GPUs, with a Python API, custom attention kernels, in-flight batching, FP8 and FP4 quantization, and multi-GPU parallelism.
Text Generation Inference (TGI)
Hugging Face · huggingface.co
Text Generation Inference is Hugging Face's Rust and Python server for LLMs, with continuous batching, tensor parallelism, quantization, and an OpenAI-compatible Messages API, and it previously powered Hugging Face's own inference services.
Hugging Face put TGI in maintenance mode and archived its GitHub repository, and its documentation now recommends vLLM, SGLang, llama.cpp, or MLX.

TextGen (formerly text-generation-webui)
github.com
TextGen is a desktop app and web interface for local LLMs that runs GGUF models through llama.cpp, adds optional ExLlamaV3 and Transformers backends, and serves OpenAI- and Anthropic-compatible APIs.
The project formerly known as text-generation-webui now ships as TextGen from the oobabooga/textgen repository.

TokenSpeed
LightSeek Foundation · lightseek.org
TokenSpeed is an inference engine for agentic workloads, launched in 2026 by the nonprofit LightSeek Foundation, that separates a C++ control plane from a Python execution plane and runs on NVIDIA and AMD GPUs.
vLLM
vllm.ai
vLLM is an open-source inference and serving engine for LLMs built around PagedAttention for KV cache memory management, with continuous batching, an OpenAI-compatible server, and support for NVIDIA, AMD, TPU, and other accelerators.

Xinference
Xorbits · xinference.io
Xinference is a platform from Xorbits that serves language, embedding, image, and audio models through an OpenAI-compatible API, using vLLM, SGLang, llama.cpp, MLX, or Transformers as backends on one machine or a cluster.
How to choose
- 01Hardware support
- Each engine supports NVIDIA, AMD, Apple Silicon, CPUs, NPUs, and other accelerators to a different degree.
- 02Throughput features
- Continuous batching, paged or prefix-aware KV caching, speculative decoding, and chunked prefill raise the number of requests one GPU can serve.
- 03Model formats and quantization
- Support for Hugging Face safetensors, GGUF, EXL3, and schemes such as FP8, FP4, INT4, or AWQ decides which checkpoints run without conversion.
- 04Scale-out
- Tensor, pipeline, and expert parallelism, plus disaggregated prefill and decode, matter for large models and multi-node clusters.
- 05API and integration
- An OpenAI-compatible server, a metrics endpoint, and Kubernetes support simplify placing the engine behind a gateway.
- 06License and governance
- Permissive licenses such as Apache-2.0 and MIT differ from AGPL-3.0 or vendor licenses, and foundation-hosted projects differ from single-company ones.
Frequently asked questions
What is an inference engine in LLM?
- An LLM inference engine is the runtime that executes a trained model to produce tokens. It loads weights into memory, schedules incoming requests into batches, stores attention keys and values in a KV cache, and streams output over an API. vLLM, SGLang, TensorRT-LLM, and llama.cpp are inference engines; Ollama and LM Studio package engines into local tools.
What is the difference between vLLM and Ollama?
- vLLM is a GPU serving engine built for many concurrent users, using PagedAttention and continuous batching to raise throughput on data center hardware. Ollama is a local tool that downloads and runs quantized models on a single machine through a CLI, a desktop app, and a local API. vLLM suits shared production endpoints, and Ollama suits development and personal use.
vLLM vs SGLang: which is better?
- Neither engine is better for every workload. Both are Apache-2.0 GPU serving engines with OpenAI-compatible servers and broad model support. SGLang's RadixAttention reuses cached prefixes across requests, which helps agent and multi-turn workloads with shared context, while vLLM supports a wide range of hardware backends. Benchmark both on your model, hardware, and traffic before choosing.
llama.cpp vs Ollama: what is the difference?
- llama.cpp is a C/C++ library and server that runs GGUF models on CPUs, Apple Silicon, and GPUs, and it exposes many low-level options. Ollama is a separate tool that wraps model download, management, and serving behind simpler commands and a local API. Developers who want fine-grained control often use llama.cpp directly, while those who want a quick setup often start with Ollama.
TensorRT-LLM vs vLLM: what is the difference?
- TensorRT-LLM is NVIDIA's open-source library for optimizing inference on NVIDIA GPUs, with custom attention kernels and FP8 and FP4 quantization tuned for that hardware. vLLM is a community project that runs on NVIDIA, AMD, TPU, and other accelerators and adds new models quickly. TensorRT-LLM suits NVIDIA-only fleets tuned for efficiency, and vLLM suits mixed hardware and fast model adoption.
Is Hugging Face TGI still maintained?
- Text Generation Inference is in maintenance mode, and its GitHub repository is archived, so it no longer receives new features. Hugging Face's documentation recommends vLLM or SGLang for GPU serving and local engines such as llama.cpp or MLX for running models on personal hardware. Existing TGI deployments keep working, but new projects are better served by an actively developed engine.
Related guides
LLM inference and serving hubOther categories
- Model providers
- Inference providers
- GPU clouds
- Vector databases
- Embedding and reranking models
- Document parsing and data ingestion
- Fine-tuning and post-training
- Agent frameworks
- Agent infrastructure
- Web search APIs
- MCP platforms
- Voice AI
- AI gateways
- LLM observability and evaluation
- Guardrails and AI security
- AI coding tools
Product details change quickly; check each vendor’s site for current features, licensing and pricing.