---
title: "Serving engines"
description: "Compare open-source LLM inference engines and serving frameworks, including vLLM, SGLang, TensorRT-LLM, llama.cpp, and Ollama, for local and production use."
type: "platform-category"
canonical_url: "https://theaiengineering.com/platforms/serving-engines"
markdown_url: "https://theaiengineering.com/platforms/serving-engines.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
citation: "The AI Engineering Editors. \"Serving engines.\" The AI Engineering. https://theaiengineering.com/platforms/serving-engines"
---

# Serving engines

An LLM serving engine is software that loads model weights onto GPUs or CPUs and generates tokens in response to requests, managing batching, KV cache memory, and an HTTP API. The engine determines how many concurrent users one machine can serve and how quickly each response starts.

Production engines such as vLLM, SGLang, TensorRT-LLM, and TokenSpeed target multi-user GPU serving with continuous batching, prefix caching, quantization, and multi-GPU parallelism, while NVIDIA Dynamo, llm-d, and KServe coordinate those engines across clusters. Local tools such as llama.cpp, Ollama, LM Studio, and MLX-LM focus on running quantized models on a single laptop or workstation.

Most engines expose an OpenAI-compatible endpoint, so applications can move between them with few code changes. The choice depends mostly on hardware, model format, and whether one person or many users share the deployment. Hugging Face's TGI is now in maintenance mode, so new deployments usually start with another engine.

## Serving engines compared

| Platform | Open source | Deployment | Best for |
| --- | --- | --- | --- |
| [BentoML](https://bentoml.com) | Yes (Apache-2.0) | Self-hosted, Library | Packaging custom inference code as production APIs |
| [Docker Model Runner](https://docs.docker.com/ai/model-runner/) | Yes (Apache-2.0) | Local, CLI, Desktop app | Teams that already build and ship with Docker |
| [ExLlamaV3](https://github.com/turboderp-org/exllamav3) | Yes (MIT) | Local, Library | Large quantized models on consumer NVIDIA GPUs |
| [exo](https://exolabs.net) | Yes (Apache-2.0) | Local, Desktop app | Running large models across several home devices |
| [Foundry Local](https://www.foundrylocal.ai) | No | Local, Library, CLI | Shipping local inference inside Windows and cross-platform apps |
| [Jan](https://jan.ai) | Yes (Apache-2.0) | Desktop app, Local | An open-source desktop chat app for local models |
| [KoboldCpp](https://github.com/LostRuins/koboldcpp) | Yes (AGPL-3.0) | Local, Desktop app | Zero-install local models for chat and writing |
| [KServe](https://kserve.github.io/website/) | Yes (Apache-2.0) | Self-hosted | Standardized model serving on Kubernetes |
| [KTransformers](https://github.com/kvcache-ai/ktransformers) | Yes (Apache-2.0) | Self-hosted, Library | Large MoE models on a single workstation |
| [Lemonade](https://lemonade-server.ai) | Yes (Apache-2.0) | Local, Desktop app | Local models on AMD Ryzen AI and Radeon PCs |
| [LightLLM](https://github.com/ModelTC/LightLLM) | Yes (Apache-2.0) | Self-hosted, Library | A compact Python codebase for serving research |
| [LiteRT-LM](https://ai.google.dev/edge/litert-lm) | Yes (Apache-2.0) | Local, Library, CLI | On-device Gemma and small models in mobile and web apps |
| [llama.cpp](https://github.com/ggml-org/llama.cpp) | Yes (MIT) | Local, Self-hosted, Library, CLI | Quantized models on laptops, desktops, and edge devices |
| [llamafile](https://github.com/mozilla-ai/llamafile) | Yes (Apache-2.0) | Local, CLI | Distributing a model as one portable executable |
| [llm-d](https://llm-d.ai) | Yes (Apache-2.0) | Self-hosted | Running vLLM at scale on Kubernetes |
| [LM Studio](https://lmstudio.ai) | No | Desktop app, Local, CLI | A graphical app for local models on one machine |
| [LMDeploy](https://github.com/InternLM/lmdeploy) | Yes (Apache-2.0) | Self-hosted, Library | Quantized LLM serving on NVIDIA GPUs |
| [LocalAI](https://localai.io) | Yes (MIT) | Local, Self-hosted | A self-hosted drop-in for several OpenAI APIs |
| [mistral.rs](https://github.com/EricLBuehler/mistral.rs) | Yes (MIT) | Local, Self-hosted, Library | Rust applications that embed local inference |
| [MLC LLM](https://llm.mlc.ai) | Yes (Apache-2.0) | Local, Self-hosted, Library | One model build for phones, browsers, and GPUs |
| [MLX-LM](https://github.com/ml-explore/mlx-lm) | Yes (MIT) | Local, Library, CLI | Running and fine-tuning models on Apple Silicon Macs |
| [Modular MAX](https://www.modular.com/max) | No | Self-hosted, Library | Portable serving across NVIDIA, AMD, and Apple hardware |
| [NVIDIA Dynamo](https://developer.nvidia.com/dynamo) | Yes (Apache-2.0) | Self-hosted | Multi-node serving across large GPU clusters |
| [NVIDIA Dynamo-Triton (formerly Triton Inference Server)](https://developer.nvidia.com/dynamo-triton) | Yes (BSD-3-Clause) | Self-hosted | Serving mixed model types from one server |
| [Ollama](https://ollama.com) | Yes (MIT) | Local, Self-hosted, CLI, Desktop app, Managed cloud | Running open models locally with minimal setup |
| [RamaLama](https://ramalama.ai) | Yes (MIT) | Local, CLI | Running local models in containers with Podman or Docker |
| [SGLang](https://www.sglang.io) | Yes (Apache-2.0) | Self-hosted, Library | Workloads with shared prefixes and large MoE models |
| [Sonar (formerly Aphrodite Engine)](https://sonar.dphn.ai) | Yes (AGPL-3.0) | Self-hosted, Library | vLLM-style serving with extra samplers and quant formats |
| [TensorRT-LLM](https://developer.nvidia.com/tensorrt-llm) | Yes (Apache-2.0) | Self-hosted, Library | Maximizing performance on NVIDIA GPU fleets |
| [Text Generation Inference (TGI)](https://huggingface.co/docs/text-generation-inference/index) | Yes (Apache-2.0) | Self-hosted | Maintaining existing TGI deployments |
| [TextGen (formerly text-generation-webui)](https://github.com/oobabooga/textgen) | Yes (AGPL-3.0) | Desktop app, Local | Experimenting with local models across several backends |
| [TokenSpeed](https://lightseek.org) | Yes (MIT) | Self-hosted, Library | Agentic workloads on recent NVIDIA and AMD GPUs |
| [vLLM](https://vllm.ai) | Yes (Apache-2.0) | Self-hosted, Library | High-throughput multi-user serving on GPUs |
| [Xinference](https://xinference.io) | Yes (Apache-2.0) | Self-hosted, Local | One API over several engines and model types |

## Profiles

### BentoML

BentoML is a Python framework for packaging models and inference code into APIs, job queues, and multi-model pipelines, with integrations for vLLM and other engines and container images for deployment anywhere.

Modular acquired BentoML in February 2026, and the BentoML framework remains open source under Apache-2.0.

- Company: Modular
- Website: https://bentoml.com
- GitHub: https://github.com/bentoml/BentoML
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python
- Best for: Packaging custom inference code as production APIs
- Permalink: https://theaiengineering.com/platforms/serving-engines#bentoml

### Docker Model Runner

Docker Model Runner pulls models as OCI artifacts from Docker Hub or other registries and runs them locally through Docker Desktop or Docker Engine with llama.cpp, vLLM, or SGLang behind an OpenAI-compatible API.

- Company: Docker
- Website: https://docs.docker.com/ai/model-runner/
- GitHub: https://github.com/docker/model-runner
- Open source: Yes (Apache-2.0)
- Deployment: Local, CLI, Desktop app
- Languages: Go
- Best for: Teams that already build and ship with Docker
- Permalink: https://theaiengineering.com/platforms/serving-engines#docker-model-runner

### ExLlamaV3

ExLlamaV3 is an MIT-licensed quantization and inference library for running LLMs on consumer NVIDIA GPUs with its EXL3 format; it succeeds ExLlamaV2, whose last release came in July 2025.

- Website: https://github.com/turboderp-org/exllamav3
- GitHub: https://github.com/turboderp-org/exllamav3
- Open source: Yes (MIT)
- Deployment: Local, Library
- Languages: Python
- Best for: Large quantized models on consumer NVIDIA GPUs
- Permalink: https://theaiengineering.com/platforms/serving-engines#exllamav3

### exo

exo connects Macs and other devices on a local network into one cluster for running models too large for a single machine, splitting each model across devices and exposing OpenAI-, Claude-, and Ollama-compatible APIs.

- Company: Exo Labs
- Website: https://exolabs.net
- GitHub: https://github.com/exo-explore/exo
- Open source: Yes (Apache-2.0)
- Deployment: Local, Desktop app
- Languages: Python, Swift
- Best for: Running large models across several home devices
- Permalink: https://theaiengineering.com/platforms/serving-engines#exo

### Foundry Local

Foundry Local is Microsoft's on-device inference runtime that downloads optimized models and runs them with ONNX Runtime on CPUs, GPUs, and NPUs, shipping inside applications through SDKs with no Azure subscription required.

- Company: Microsoft
- Website: https://www.foundrylocal.ai
- GitHub: https://github.com/microsoft/foundry-local
- Open source: No
- Deployment: Local, Library, CLI
- Languages: C++, C#, JavaScript, Python, Rust
- Best for: Shipping local inference inside Windows and cross-platform apps
- Permalink: https://theaiengineering.com/platforms/serving-engines#foundry-local

### Jan

Jan is an Apache-2.0 desktop app from Menlo Research that runs open models locally through llama.cpp, connects to cloud model APIs, and exposes a local OpenAI-compatible server for other applications.

- Company: Menlo Research
- Website: https://jan.ai
- GitHub: https://github.com/janhq/jan
- Open source: Yes (Apache-2.0)
- Deployment: Desktop app, Local
- Languages: TypeScript, Rust
- Best for: An open-source desktop chat app for local models
- Permalink: https://theaiengineering.com/platforms/serving-engines#jan

### KoboldCpp

KoboldCpp is a single-file program built on llama.cpp that runs GGUF models locally with the bundled KoboldAI Lite interface and serves KoboldAI, OpenAI-compatible, and Ollama-compatible APIs, plus image and speech generation.

- Website: https://github.com/LostRuins/koboldcpp
- GitHub: https://github.com/LostRuins/koboldcpp
- Open source: Yes (AGPL-3.0)
- Deployment: Local, Desktop app
- Languages: C++, Python
- Best for: Zero-install local models for chat and writing
- Permalink: https://theaiengineering.com/platforms/serving-engines#koboldcpp

### KServe

KServe is a CNCF incubating project for serving predictive and generative models on Kubernetes, providing InferenceService resources, autoscaling, OpenAI-compatible endpoints, and vLLM and llm-d backends for LLMs.

- Website: https://kserve.github.io/website/
- GitHub: https://github.com/kserve/kserve
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted
- Languages: Go, Python
- Best for: Standardized model serving on Kubernetes
- Permalink: https://theaiengineering.com/platforms/serving-engines#kserve

### KTransformers

KTransformers is a framework for heterogeneous CPU and GPU inference and fine-tuning that keeps parts of large mixture-of-experts models in system memory, so they can run on machines with limited GPU memory.

- Website: https://github.com/kvcache-ai/ktransformers
- GitHub: https://github.com/kvcache-ai/ktransformers
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python, C++
- Best for: Large MoE models on a single workstation
- Permalink: https://theaiengineering.com/platforms/serving-engines#ktransformers

### Lemonade

Lemonade is a local AI server that runs language, speech, and image models on PC GPUs and NPUs, with optimizations from AMD engineers for Ryzen AI and Radeon hardware and OpenAI-, Anthropic-, and Ollama-compatible APIs.

- Website: https://lemonade-server.ai
- GitHub: https://github.com/lemonade-sdk/lemonade
- Open source: Yes (Apache-2.0)
- Deployment: Local, Desktop app
- Languages: C++, Python
- Best for: Local models on AMD Ryzen AI and Radeon PCs
- Permalink: https://theaiengineering.com/platforms/serving-engines#lemonade

### LightLLM

LightLLM is a Python-based LLM inference and serving framework from ModelTC with a lightweight design, token-level KV cache management, and multi-GPU support, aimed at research and production serving.

- Website: https://github.com/ModelTC/LightLLM
- GitHub: https://github.com/ModelTC/LightLLM
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python
- Best for: A compact Python codebase for serving research
- Permalink: https://theaiengineering.com/platforms/serving-engines#lightllm

### LiteRT-LM

LiteRT-LM is Google's runtime for running LLMs on device with LiteRT across mobile, web, desktop, and Raspberry Pi, with GPU acceleration, constrained decoding, and speculative decoding for models such as Gemma.

- Company: Google
- Website: https://ai.google.dev/edge/litert-lm
- GitHub: https://github.com/google-ai-edge/LiteRT-LM
- Open source: Yes (Apache-2.0)
- Deployment: Local, Library, CLI
- Languages: C++, Python, TypeScript
- Best for: On-device Gemma and small models in mobile and web apps
- Permalink: https://theaiengineering.com/platforms/serving-engines#litert-lm

### llama.cpp

llama.cpp is an MIT-licensed C/C++ inference library and server that runs quantized models in the GGUF format on CPUs, Apple Silicon, and GPUs; its maintainers at ggml.ai joined Hugging Face in February 2026.

- Website: https://github.com/ggml-org/llama.cpp
- GitHub: https://github.com/ggml-org/llama.cpp
- Open source: Yes (MIT)
- Deployment: Local, Self-hosted, Library, CLI
- Languages: C, C++
- Best for: Quantized models on laptops, desktops, and edge devices
- Permalink: https://theaiengineering.com/platforms/serving-engines#llama-cpp

### llamafile

llamafile is a Mozilla.ai project that bundles model weights and a llama.cpp-based runtime into a single executable that runs on macOS, Windows, Linux, and BSD without installation, using Cosmopolitan Libc.

- Company: Mozilla.ai
- Website: https://github.com/mozilla-ai/llamafile
- GitHub: https://github.com/mozilla-ai/llamafile
- Open source: Yes (Apache-2.0)
- Deployment: Local, CLI
- Languages: C++, C
- Best for: Distributing a model as one portable executable
- Permalink: https://theaiengineering.com/platforms/serving-engines#llamafile

### llm-d

llm-d is a Kubernetes-native distributed inference stack and CNCF sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, that adds cache-aware routing and disaggregated serving above vLLM or SGLang.

- Website: https://llm-d.ai
- GitHub: https://github.com/llm-d/llm-d
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted
- Languages: Python, Go
- Best for: Running vLLM at scale on Kubernetes
- Permalink: https://theaiengineering.com/platforms/serving-engines#llm-d

### LM Studio

LM Studio is a desktop app for macOS, Windows, and Linux that downloads and runs local models with llama.cpp or Apple MLX, serves them on OpenAI-compatible endpoints, and includes Bionic, an agent for documents and code.

- Website: https://lmstudio.ai
- GitHub: https://github.com/lmstudio-ai
- Open source: No
- Deployment: Desktop app, Local, CLI
- Languages: Python, TypeScript
- Best for: A graphical app for local models on one machine
- Permalink: https://theaiengineering.com/platforms/serving-engines#lm-studio

### LMDeploy

LMDeploy, from the InternLM team at Shanghai AI Laboratory, is a toolkit for compressing, deploying, and serving LLMs with its C++ TurboMind engine or a PyTorch engine, with quantization and an OpenAI-compatible server.

- Website: https://github.com/InternLM/lmdeploy
- GitHub: https://github.com/InternLM/lmdeploy
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python, C++
- Best for: Quantized LLM serving on NVIDIA GPUs
- Permalink: https://theaiengineering.com/platforms/serving-engines#lmdeploy

### LocalAI

LocalAI is an MIT-licensed, OpenAI-compatible server that runs language, vision, speech, image, and video models on local hardware with or without a GPU, switching between backends such as llama.cpp, vLLM, and diffusers.

- Website: https://localai.io
- GitHub: https://github.com/mudler/LocalAI
- Open source: Yes (MIT)
- Deployment: Local, Self-hosted
- Languages: Go
- Best for: A self-hosted drop-in for several OpenAI APIs
- Permalink: https://theaiengineering.com/platforms/serving-engines#localai

### mistral.rs

mistral.rs is an LLM inference engine written in Rust with in-situ quantization, multimodal model support, an OpenAI-compatible server, and Python and Rust APIs for embedding inference in applications.

- Website: https://github.com/EricLBuehler/mistral.rs
- GitHub: https://github.com/EricLBuehler/mistral.rs
- Open source: Yes (MIT)
- Deployment: Local, Self-hosted, Library
- Languages: Rust, Python
- Best for: Rust applications that embed local inference
- Permalink: https://theaiengineering.com/platforms/serving-engines#mistral-rs

### MLC LLM

MLC LLM is a compiler-based engine that uses Apache TVM to compile models for GPUs, Apple Silicon, iOS, Android, and web browsers through WebLLM, exposing an OpenAI-compatible API on each platform.

- Website: https://llm.mlc.ai
- GitHub: https://github.com/mlc-ai/mlc-llm
- Open source: Yes (Apache-2.0)
- Deployment: Local, Self-hosted, Library
- Languages: Python, C++
- Best for: One model build for phones, browsers, and GPUs
- Permalink: https://theaiengineering.com/platforms/serving-engines#mlc-llm

### MLX-LM

MLX-LM is an MIT-licensed Python package from Apple's MLX team for running and fine-tuning LLMs on Apple Silicon with the MLX array framework, including quantization, LoRA training, and an OpenAI-compatible HTTP server.

- Company: Apple
- Website: https://github.com/ml-explore/mlx-lm
- GitHub: https://github.com/ml-explore/mlx-lm
- Open source: Yes (MIT)
- Deployment: Local, Library, CLI
- Languages: Python
- Best for: Running and fine-tuning models on Apple Silicon Macs
- Permalink: https://theaiengineering.com/platforms/serving-engines#mlx-lm

### Modular MAX

MAX is Modular's serving and modeling framework that runs the same package on NVIDIA, AMD, and Apple silicon, exposes OpenAI-compatible APIs, and uses kernels written in Mojo; its source is partly Apache-2.0 and partly under the MAX license.

- Company: Modular
- Website: https://www.modular.com/max
- GitHub: https://github.com/modular/modular
- Open source: No
- Deployment: Self-hosted, Library
- Languages: Python, Mojo
- Best for: Portable serving across NVIDIA, AMD, and Apple hardware
- Permalink: https://theaiengineering.com/platforms/serving-engines#modular-max

### NVIDIA Dynamo

NVIDIA Dynamo is an open-source orchestration layer for datacenter-scale inference that coordinates vLLM, SGLang, or TensorRT-LLM across nodes, adding disaggregated prefill and decode, KV-aware routing, multi-tier KV caching, and autoscaling.

- Company: NVIDIA
- Website: https://developer.nvidia.com/dynamo
- GitHub: https://github.com/ai-dynamo/dynamo
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted
- Languages: Rust, Python
- Best for: Multi-node serving across large GPU clusters
- Permalink: https://theaiengineering.com/platforms/serving-engines#nvidia-dynamo

### NVIDIA Dynamo-Triton (formerly Triton Inference Server)

Dynamo-Triton is NVIDIA's open-source inference server for deploying models from TensorRT, TensorRT-LLM, PyTorch, ONNX, vLLM, and other backends, with dynamic batching, model ensembles, and HTTP and gRPC endpoints.

NVIDIA renamed Triton Inference Server to Dynamo-Triton, and the GitHub repository keeps the triton-inference-server name.

- Company: NVIDIA
- Website: https://developer.nvidia.com/dynamo-triton
- GitHub: https://github.com/triton-inference-server/server
- Open source: Yes (BSD-3-Clause)
- Deployment: Self-hosted
- Languages: C++, Python
- Best for: Serving mixed model types from one server
- Permalink: https://theaiengineering.com/platforms/serving-engines#dynamo-triton

### Ollama

Ollama is an MIT-licensed tool for downloading and running open models on macOS, Windows, and Linux through a CLI, a desktop app, and a local REST API, with an option to run larger models on Ollama's cloud.

- Website: https://ollama.com
- GitHub: https://github.com/ollama/ollama
- Open source: Yes (MIT)
- Deployment: Local, Self-hosted, CLI, Desktop app, Managed cloud
- Languages: Go
- Best for: Running open models locally with minimal setup
- Permalink: https://theaiengineering.com/platforms/serving-engines#ollama

### RamaLama

RamaLama is a command-line tool from the containers community that runs AI models inside OCI containers, detecting the host's GPU and pulling a matching llama.cpp or vLLM image so models run in isolation.

- Website: https://ramalama.ai
- GitHub: https://github.com/containers/ramalama
- Open source: Yes (MIT)
- Deployment: Local, CLI
- Languages: Python
- Best for: Running local models in containers with Podman or Docker
- Permalink: https://theaiengineering.com/platforms/serving-engines#ramalama

### SGLang

SGLang is an open-source serving framework for language and multimodal models that uses RadixAttention to reuse KV cache across requests with shared prefixes, and supports prefill-decode disaggregation and expert parallelism.

- Website: https://www.sglang.io
- GitHub: https://github.com/sgl-project/sglang
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python
- Best for: Workloads with shared prefixes and large MoE models
- Permalink: https://theaiengineering.com/platforms/serving-engines#sglang

### Sonar (formerly Aphrodite Engine)

Sonar is an inference engine based on vLLM that adds extra quantization formats, sampling methods, and kernels, and serves production traffic for the Dolphin inference network and PygmalionAI.

Aphrodite Engine was renamed Sonar and moved to the dphnAI GitHub organization, and its command-line tool still uses the aphrodite name.

- Website: https://sonar.dphn.ai
- GitHub: https://github.com/dphnAI/sonar
- Open source: Yes (AGPL-3.0)
- Deployment: Self-hosted, Library
- Languages: Python
- Best for: vLLM-style serving with extra samplers and quant formats
- Permalink: https://theaiengineering.com/platforms/serving-engines#sonar-engine

### TensorRT-LLM

TensorRT-LLM is NVIDIA's open-source library for optimizing LLM inference on NVIDIA GPUs, with a Python API, custom attention kernels, in-flight batching, FP8 and FP4 quantization, and multi-GPU parallelism.

- Company: NVIDIA
- Website: https://developer.nvidia.com/tensorrt-llm
- GitHub: https://github.com/NVIDIA/TensorRT-LLM
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python, C++
- Best for: Maximizing performance on NVIDIA GPU fleets
- Permalink: https://theaiengineering.com/platforms/serving-engines#tensorrt-llm

### Text Generation Inference (TGI)

Text Generation Inference is Hugging Face's Rust and Python server for LLMs, with continuous batching, tensor parallelism, quantization, and an OpenAI-compatible Messages API, and it previously powered Hugging Face's own inference services.

Hugging Face put TGI in maintenance mode and archived its GitHub repository, and its documentation now recommends vLLM, SGLang, llama.cpp, or MLX.

- Company: Hugging Face
- Website: https://huggingface.co/docs/text-generation-inference/index
- GitHub: https://github.com/huggingface/text-generation-inference
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted
- Languages: Rust, Python
- Best for: Maintaining existing TGI deployments
- Permalink: https://theaiengineering.com/platforms/serving-engines#text-generation-inference

### TextGen (formerly text-generation-webui)

TextGen is a desktop app and web interface for local LLMs that runs GGUF models through llama.cpp, adds optional ExLlamaV3 and Transformers backends, and serves OpenAI- and Anthropic-compatible APIs.

The project formerly known as text-generation-webui now ships as TextGen from the oobabooga/textgen repository.

- Website: https://github.com/oobabooga/textgen
- GitHub: https://github.com/oobabooga/textgen
- Open source: Yes (AGPL-3.0)
- Deployment: Desktop app, Local
- Languages: Python
- Best for: Experimenting with local models across several backends
- Permalink: https://theaiengineering.com/platforms/serving-engines#textgen

### TokenSpeed

TokenSpeed is an inference engine for agentic workloads, launched in 2026 by the nonprofit LightSeek Foundation, that separates a C++ control plane from a Python execution plane and runs on NVIDIA and AMD GPUs.

- Company: LightSeek Foundation
- Website: https://lightseek.org
- GitHub: https://github.com/lightseekorg/tokenspeed
- Open source: Yes (MIT)
- Deployment: Self-hosted, Library
- Languages: Python, C++, Rust
- Best for: Agentic workloads on recent NVIDIA and AMD GPUs
- Permalink: https://theaiengineering.com/platforms/serving-engines#tokenspeed

### vLLM

vLLM is an open-source inference and serving engine for LLMs built around PagedAttention for KV cache memory management, with continuous batching, an OpenAI-compatible server, and support for NVIDIA, AMD, TPU, and other accelerators.

- Website: https://vllm.ai
- GitHub: https://github.com/vllm-project/vllm
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Library
- Languages: Python
- Best for: High-throughput multi-user serving on GPUs
- Permalink: https://theaiengineering.com/platforms/serving-engines#vllm

### Xinference

Xinference is a platform from Xorbits that serves language, embedding, image, and audio models through an OpenAI-compatible API, using vLLM, SGLang, llama.cpp, MLX, or Transformers as backends on one machine or a cluster.

- Company: Xorbits
- Website: https://xinference.io
- GitHub: https://github.com/xorbitsai/inference
- Open source: Yes (Apache-2.0)
- Deployment: Self-hosted, Local
- Languages: Python
- Best for: One API over several engines and model types
- Permalink: https://theaiengineering.com/platforms/serving-engines#xinference

## How to choose

1. **Hardware support**: Each engine supports NVIDIA, AMD, Apple Silicon, CPUs, NPUs, and other accelerators to a different degree.
2. **Throughput features**: Continuous batching, paged or prefix-aware KV caching, speculative decoding, and chunked prefill raise the number of requests one GPU can serve.
3. **Model formats and quantization**: Support for Hugging Face safetensors, GGUF, EXL3, and schemes such as FP8, FP4, INT4, or AWQ decides which checkpoints run without conversion.
4. **Scale-out**: Tensor, pipeline, and expert parallelism, plus disaggregated prefill and decode, matter for large models and multi-node clusters.
5. **API and integration**: An OpenAI-compatible server, a metrics endpoint, and Kubernetes support simplify placing the engine behind a gateway.
6. **License and governance**: Permissive licenses such as Apache-2.0 and MIT differ from AGPL-3.0 or vendor licenses, and foundation-hosted projects differ from single-company ones.

## Frequently asked questions

### What is an inference engine in LLM?

An LLM inference engine is the runtime that executes a trained model to produce tokens. It loads weights into memory, schedules incoming requests into batches, stores attention keys and values in a KV cache, and streams output over an API. vLLM, SGLang, TensorRT-LLM, and llama.cpp are inference engines; Ollama and LM Studio package engines into local tools.

### What is the difference between vLLM and Ollama?

vLLM is a GPU serving engine built for many concurrent users, using PagedAttention and continuous batching to raise throughput on data center hardware. Ollama is a local tool that downloads and runs quantized models on a single machine through a CLI, a desktop app, and a local API. vLLM suits shared production endpoints, and Ollama suits development and personal use.

### vLLM vs SGLang: which is better?

Neither engine is better for every workload. Both are Apache-2.0 GPU serving engines with OpenAI-compatible servers and broad model support. SGLang's RadixAttention reuses cached prefixes across requests, which helps agent and multi-turn workloads with shared context, while vLLM supports a wide range of hardware backends. Benchmark both on your model, hardware, and traffic before choosing.

### llama.cpp vs Ollama: what is the difference?

llama.cpp is a C/C++ library and server that runs GGUF models on CPUs, Apple Silicon, and GPUs, and it exposes many low-level options. Ollama is a separate tool that wraps model download, management, and serving behind simpler commands and a local API. Developers who want fine-grained control often use llama.cpp directly, while those who want a quick setup often start with Ollama.

### TensorRT-LLM vs vLLM: what is the difference?

TensorRT-LLM is NVIDIA's open-source library for optimizing inference on NVIDIA GPUs, with custom attention kernels and FP8 and FP4 quantization tuned for that hardware. vLLM is a community project that runs on NVIDIA, AMD, TPU, and other accelerators and adds new models quickly. TensorRT-LLM suits NVIDIA-only fleets tuned for efficiency, and vLLM suits mixed hardware and fast model adoption.

### Is Hugging Face TGI still maintained?

Text Generation Inference is in maintenance mode, and its GitHub repository is archived, so it no longer receives new features. Hugging Face's documentation recommends vLLM or SGLang for GPU serving and local engines such as llama.cpp or MLX for running models on personal hardware. Existing TGI deployments keep working, but new projects are better served by an actively developed engine.

## Related guides

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.

Product details change quickly; check each vendor's site for current features, licensing and pricing.

## Cite this page

The AI Engineering Editors. "Serving engines." The AI Engineering. https://theaiengineering.com/platforms/serving-engines

- Title: Serving engines
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- URL: https://theaiengineering.com/platforms/serving-engines
- Markdown: https://theaiengineering.com/platforms/serving-engines.md
