Serving engines

An LLM serving engine is software that loads model weights onto GPUs or CPUs and generates tokens in response to requests, managing batching, KV cache memory, and an HTTP API. The engine determines how many concurrent users one machine can serve and how quickly each response starts.

Production engines such as vLLM, SGLang, TensorRT-LLM, and TokenSpeed target multi-user GPU serving with continuous batching, prefix caching, quantization, and multi-GPU parallelism, while NVIDIA Dynamo, llm-d, and KServe coordinate those engines across clusters. Local tools such as llama.cpp, Ollama, LM Studio, and MLX-LM focus on running quantized models on a single laptop or workstation.

Most engines expose an OpenAI-compatible endpoint, so applications can move between them with few code changes. The choice depends mostly on hardware, model format, and whether one person or many users share the deployment. Hugging Face's TGI is now in maintenance mode, so new deployments usually start with another engine.

Serving engines compared

PlatformOpen sourceDeploymentBest for
BentoMLYes (Apache-2.0)Self-hosted, LibraryPackaging custom inference code as production APIs
Docker Model RunnerYes (Apache-2.0)Local, CLI, Desktop appTeams that already build and ship with Docker
ExLlamaV3Yes (MIT)Local, LibraryLarge quantized models on consumer NVIDIA GPUs
exoYes (Apache-2.0)Local, Desktop appRunning large models across several home devices
Foundry LocalNoLocal, Library, CLIShipping local inference inside Windows and cross-platform apps
JanYes (Apache-2.0)Desktop app, LocalAn open-source desktop chat app for local models
KoboldCppYes (AGPL-3.0)Local, Desktop appZero-install local models for chat and writing
KServeYes (Apache-2.0)Self-hostedStandardized model serving on Kubernetes
KTransformersYes (Apache-2.0)Self-hosted, LibraryLarge MoE models on a single workstation
LemonadeYes (Apache-2.0)Local, Desktop appLocal models on AMD Ryzen AI and Radeon PCs
LightLLMYes (Apache-2.0)Self-hosted, LibraryA compact Python codebase for serving research
LiteRT-LMYes (Apache-2.0)Local, Library, CLIOn-device Gemma and small models in mobile and web apps
llama.cppYes (MIT)Local, Self-hosted, Library, CLIQuantized models on laptops, desktops, and edge devices
llamafileYes (Apache-2.0)Local, CLIDistributing a model as one portable executable
llm-dYes (Apache-2.0)Self-hostedRunning vLLM at scale on Kubernetes
LM StudioNoDesktop app, Local, CLIA graphical app for local models on one machine
LMDeployYes (Apache-2.0)Self-hosted, LibraryQuantized LLM serving on NVIDIA GPUs
LocalAIYes (MIT)Local, Self-hostedA self-hosted drop-in for several OpenAI APIs
mistral.rsYes (MIT)Local, Self-hosted, LibraryRust applications that embed local inference
MLC LLMYes (Apache-2.0)Local, Self-hosted, LibraryOne model build for phones, browsers, and GPUs
MLX-LMYes (MIT)Local, Library, CLIRunning and fine-tuning models on Apple Silicon Macs
Modular MAXNoSelf-hosted, LibraryPortable serving across NVIDIA, AMD, and Apple hardware
NVIDIA DynamoYes (Apache-2.0)Self-hostedMulti-node serving across large GPU clusters
NVIDIA Dynamo-Triton (formerly Triton Inference Server)Yes (BSD-3-Clause)Self-hostedServing mixed model types from one server
OllamaYes (MIT)Local, Self-hosted, CLI, Desktop app, Managed cloudRunning open models locally with minimal setup
RamaLamaYes (MIT)Local, CLIRunning local models in containers with Podman or Docker
SGLangYes (Apache-2.0)Self-hosted, LibraryWorkloads with shared prefixes and large MoE models
Sonar (formerly Aphrodite Engine)Yes (AGPL-3.0)Self-hosted, LibraryvLLM-style serving with extra samplers and quant formats
TensorRT-LLMYes (Apache-2.0)Self-hosted, LibraryMaximizing performance on NVIDIA GPU fleets
Text Generation Inference (TGI)Yes (Apache-2.0)Self-hostedMaintaining existing TGI deployments
TextGen (formerly text-generation-webui)Yes (AGPL-3.0)Desktop app, LocalExperimenting with local models across several backends
TokenSpeedYes (MIT)Self-hosted, LibraryAgentic workloads on recent NVIDIA and AMD GPUs
vLLMYes (Apache-2.0)Self-hosted, LibraryHigh-throughput multi-user serving on GPUs
XinferenceYes (Apache-2.0)Self-hosted, LocalOne API over several engines and model types

Profiles

  • BentoML

    Modular · bentoml.com

    BentoML is a Python framework for packaging models and inference code into APIs, job queues, and multi-model pipelines, with integrations for vLLM and other engines and container images for deployment anywhere.

    Modular acquired BentoML in February 2026, and the BentoML framework remains open source under Apache-2.0.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
  • Docker Model Runner logo

    Docker Model Runner

    Docker · docs.docker.com

    Docker Model Runner pulls models as OCI artifacts from Docker Hub or other registries and runs them locally through Docker Desktop or Docker Engine with llama.cpp, vLLM, or SGLang behind an OpenAI-compatible API.

    • Open source · Apache-2.0
    • Local
    • CLI
    • Desktop app
    • Go
  • ExLlamaV3 logo

    ExLlamaV3

    github.com

    ExLlamaV3 is an MIT-licensed quantization and inference library for running LLMs on consumer NVIDIA GPUs with its EXL3 format; it succeeds ExLlamaV2, whose last release came in July 2025.

    • Open source · MIT
    • Local
    • Library
    • Python
  • exo logo

    exo

    Exo Labs · exolabs.net

    exo connects Macs and other devices on a local network into one cluster for running models too large for a single machine, splitting each model across devices and exposing OpenAI-, Claude-, and Ollama-compatible APIs.

    • Open source · Apache-2.0
    • Local
    • Desktop app
    • Python
    • Swift
  • Foundry Local logo

    Foundry Local

    Microsoft · foundrylocal.ai

    Foundry Local is Microsoft's on-device inference runtime that downloads optimized models and runs them with ONNX Runtime on CPUs, GPUs, and NPUs, shipping inside applications through SDKs with no Azure subscription required.

    • Local
    • Library
    • CLI
    • C++
    • C#
    • JavaScript
  • Jan logo

    Jan

    Menlo Research · jan.ai

    Jan is an Apache-2.0 desktop app from Menlo Research that runs open models locally through llama.cpp, connects to cloud model APIs, and exposes a local OpenAI-compatible server for other applications.

    • Open source · Apache-2.0
    • Desktop app
    • Local
    • TypeScript
    • Rust
  • KoboldCpp logo

    KoboldCpp

    github.com

    KoboldCpp is a single-file program built on llama.cpp that runs GGUF models locally with the bundled KoboldAI Lite interface and serves KoboldAI, OpenAI-compatible, and Ollama-compatible APIs, plus image and speech generation.

    • Open source · AGPL-3.0
    • Local
    • Desktop app
    • C++
    • Python
  • KServe logo

    KServe

    kserve.github.io

    KServe is a CNCF incubating project for serving predictive and generative models on Kubernetes, providing InferenceService resources, autoscaling, OpenAI-compatible endpoints, and vLLM and llm-d backends for LLMs.

    • Open source · Apache-2.0
    • Self-hosted
    • Go
    • Python
  • KTransformers logo

    KTransformers

    github.com

    KTransformers is a framework for heterogeneous CPU and GPU inference and fine-tuning that keeps parts of large mixture-of-experts models in system memory, so they can run on machines with limited GPU memory.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
    • C++
  • Lemonade logo

    Lemonade

    lemonade-server.ai

    Lemonade is a local AI server that runs language, speech, and image models on PC GPUs and NPUs, with optimizations from AMD engineers for Ryzen AI and Radeon hardware and OpenAI-, Anthropic-, and Ollama-compatible APIs.

    • Open source · Apache-2.0
    • Local
    • Desktop app
    • C++
    • Python
  • LightLLM logo

    LightLLM

    github.com

    LightLLM is a Python-based LLM inference and serving framework from ModelTC with a lightweight design, token-level KV cache management, and multi-GPU support, aimed at research and production serving.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
  • LiteRT-LM logo

    LiteRT-LM

    Google · ai.google.dev

    LiteRT-LM is Google's runtime for running LLMs on device with LiteRT across mobile, web, desktop, and Raspberry Pi, with GPU acceleration, constrained decoding, and speculative decoding for models such as Gemma.

    • Open source · Apache-2.0
    • Local
    • Library
    • CLI
    • C++
    • Python
    • TypeScript
  • llama.cpp logo

    llama.cpp

    github.com

    llama.cpp is an MIT-licensed C/C++ inference library and server that runs quantized models in the GGUF format on CPUs, Apple Silicon, and GPUs; its maintainers at ggml.ai joined Hugging Face in February 2026.

    • Open source · MIT
    • Local
    • Self-hosted
    • Library
    • CLI
    • C
    • C++
  • llamafile logo

    llamafile

    Mozilla.ai · github.com

    llamafile is a Mozilla.ai project that bundles model weights and a llama.cpp-based runtime into a single executable that runs on macOS, Windows, Linux, and BSD without installation, using Cosmopolitan Libc.

    • Open source · Apache-2.0
    • Local
    • CLI
    • C++
    • C
  • llm-d logo

    llm-d

    llm-d.ai

    llm-d is a Kubernetes-native distributed inference stack and CNCF sandbox project, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, that adds cache-aware routing and disaggregated serving above vLLM or SGLang.

    • Open source · Apache-2.0
    • Self-hosted
    • Python
    • Go
  • LM Studio

    lmstudio.ai

    LM Studio is a desktop app for macOS, Windows, and Linux that downloads and runs local models with llama.cpp or Apple MLX, serves them on OpenAI-compatible endpoints, and includes Bionic, an agent for documents and code.

    • Desktop app
    • Local
    • CLI
    • Python
    • TypeScript
  • LMDeploy logo

    LMDeploy

    github.com

    LMDeploy, from the InternLM team at Shanghai AI Laboratory, is a toolkit for compressing, deploying, and serving LLMs with its C++ TurboMind engine or a PyTorch engine, with quantization and an OpenAI-compatible server.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
    • C++
  • LocalAI logo

    LocalAI

    localai.io

    LocalAI is an MIT-licensed, OpenAI-compatible server that runs language, vision, speech, image, and video models on local hardware with or without a GPU, switching between backends such as llama.cpp, vLLM, and diffusers.

    • Open source · MIT
    • Local
    • Self-hosted
    • Go
  • mistral.rs logo

    mistral.rs

    github.com

    mistral.rs is an LLM inference engine written in Rust with in-situ quantization, multimodal model support, an OpenAI-compatible server, and Python and Rust APIs for embedding inference in applications.

    • Open source · MIT
    • Local
    • Self-hosted
    • Library
    • Rust
    • Python
  • MLC LLM logo

    MLC LLM

    llm.mlc.ai

    MLC LLM is a compiler-based engine that uses Apache TVM to compile models for GPUs, Apple Silicon, iOS, Android, and web browsers through WebLLM, exposing an OpenAI-compatible API on each platform.

    • Open source · Apache-2.0
    • Local
    • Self-hosted
    • Library
    • Python
    • C++
  • MLX-LM logo

    MLX-LM

    Apple · github.com

    MLX-LM is an MIT-licensed Python package from Apple's MLX team for running and fine-tuning LLMs on Apple Silicon with the MLX array framework, including quantization, LoRA training, and an OpenAI-compatible HTTP server.

    • Open source · MIT
    • Local
    • Library
    • CLI
    • Python
  • Modular MAX logo

    Modular MAX

    Modular · modular.com

    MAX is Modular's serving and modeling framework that runs the same package on NVIDIA, AMD, and Apple silicon, exposes OpenAI-compatible APIs, and uses kernels written in Mojo; its source is partly Apache-2.0 and partly under the MAX license.

    • Self-hosted
    • Library
    • Python
    • Mojo
  • NVIDIA Dynamo logo

    NVIDIA Dynamo

    NVIDIA · developer.nvidia.com

    NVIDIA Dynamo is an open-source orchestration layer for datacenter-scale inference that coordinates vLLM, SGLang, or TensorRT-LLM across nodes, adding disaggregated prefill and decode, KV-aware routing, multi-tier KV caching, and autoscaling.

    • Open source · Apache-2.0
    • Self-hosted
    • Rust
    • Python
  • NVIDIA Dynamo-Triton (formerly Triton Inference Server) logo

    NVIDIA Dynamo-Triton (formerly Triton Inference Server)

    NVIDIA · developer.nvidia.com

    Dynamo-Triton is NVIDIA's open-source inference server for deploying models from TensorRT, TensorRT-LLM, PyTorch, ONNX, vLLM, and other backends, with dynamic batching, model ensembles, and HTTP and gRPC endpoints.

    NVIDIA renamed Triton Inference Server to Dynamo-Triton, and the GitHub repository keeps the triton-inference-server name.

    • Open source · BSD-3-Clause
    • Self-hosted
    • C++
    • Python
  • Ollama

    ollama.com

    Ollama is an MIT-licensed tool for downloading and running open models on macOS, Windows, and Linux through a CLI, a desktop app, and a local REST API, with an option to run larger models on Ollama's cloud.

    • Open source · MIT
    • Local
    • Self-hosted
    • CLI
    • Desktop app
    • Managed cloud
    • Go
  • RamaLama logo

    RamaLama

    ramalama.ai

    RamaLama is a command-line tool from the containers community that runs AI models inside OCI containers, detecting the host's GPU and pulling a matching llama.cpp or vLLM image so models run in isolation.

    • Open source · MIT
    • Local
    • CLI
    • Python
  • SGLang logo

    SGLang

    sglang.io

    SGLang is an open-source serving framework for language and multimodal models that uses RadixAttention to reuse KV cache across requests with shared prefixes, and supports prefill-decode disaggregation and expert parallelism.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
  • Sonar (formerly Aphrodite Engine)

    sonar.dphn.ai

    Sonar is an inference engine based on vLLM that adds extra quantization formats, sampling methods, and kernels, and serves production traffic for the Dolphin inference network and PygmalionAI.

    Aphrodite Engine was renamed Sonar and moved to the dphnAI GitHub organization, and its command-line tool still uses the aphrodite name.

    • Open source · AGPL-3.0
    • Self-hosted
    • Library
    • Python
  • TensorRT-LLM logo

    TensorRT-LLM

    NVIDIA · developer.nvidia.com

    TensorRT-LLM is NVIDIA's open-source library for optimizing LLM inference on NVIDIA GPUs, with a Python API, custom attention kernels, in-flight batching, FP8 and FP4 quantization, and multi-GPU parallelism.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
    • C++
  • Text Generation Inference (TGI) logo

    Text Generation Inference (TGI)

    Hugging Face · huggingface.co

    Text Generation Inference is Hugging Face's Rust and Python server for LLMs, with continuous batching, tensor parallelism, quantization, and an OpenAI-compatible Messages API, and it previously powered Hugging Face's own inference services.

    Hugging Face put TGI in maintenance mode and archived its GitHub repository, and its documentation now recommends vLLM, SGLang, llama.cpp, or MLX.

    • Open source · Apache-2.0
    • Self-hosted
    • Rust
    • Python
  • TextGen (formerly text-generation-webui) logo

    TextGen (formerly text-generation-webui)

    github.com

    TextGen is a desktop app and web interface for local LLMs that runs GGUF models through llama.cpp, adds optional ExLlamaV3 and Transformers backends, and serves OpenAI- and Anthropic-compatible APIs.

    The project formerly known as text-generation-webui now ships as TextGen from the oobabooga/textgen repository.

    • Open source · AGPL-3.0
    • Desktop app
    • Local
    • Python
  • TokenSpeed logo

    TokenSpeed

    LightSeek Foundation · lightseek.org

    TokenSpeed is an inference engine for agentic workloads, launched in 2026 by the nonprofit LightSeek Foundation, that separates a C++ control plane from a Python execution plane and runs on NVIDIA and AMD GPUs.

    • Open source · MIT
    • Self-hosted
    • Library
    • Python
    • C++
    • Rust
  • vLLM

    vllm.ai

    vLLM is an open-source inference and serving engine for LLMs built around PagedAttention for KV cache memory management, with continuous batching, an OpenAI-compatible server, and support for NVIDIA, AMD, TPU, and other accelerators.

    • Open source · Apache-2.0
    • Self-hosted
    • Library
    • Python
  • Xinference logo

    Xinference

    Xorbits · xinference.io

    Xinference is a platform from Xorbits that serves language, embedding, image, and audio models through an OpenAI-compatible API, using vLLM, SGLang, llama.cpp, MLX, or Transformers as backends on one machine or a cluster.

    • Open source · Apache-2.0
    • Self-hosted
    • Local
    • Python

How to choose

01Hardware support
Each engine supports NVIDIA, AMD, Apple Silicon, CPUs, NPUs, and other accelerators to a different degree.
02Throughput features
Continuous batching, paged or prefix-aware KV caching, speculative decoding, and chunked prefill raise the number of requests one GPU can serve.
03Model formats and quantization
Support for Hugging Face safetensors, GGUF, EXL3, and schemes such as FP8, FP4, INT4, or AWQ decides which checkpoints run without conversion.
04Scale-out
Tensor, pipeline, and expert parallelism, plus disaggregated prefill and decode, matter for large models and multi-node clusters.
05API and integration
An OpenAI-compatible server, a metrics endpoint, and Kubernetes support simplify placing the engine behind a gateway.
06License and governance
Permissive licenses such as Apache-2.0 and MIT differ from AGPL-3.0 or vendor licenses, and foundation-hosted projects differ from single-company ones.

Frequently asked questions

What is an inference engine in LLM?

An LLM inference engine is the runtime that executes a trained model to produce tokens. It loads weights into memory, schedules incoming requests into batches, stores attention keys and values in a KV cache, and streams output over an API. vLLM, SGLang, TensorRT-LLM, and llama.cpp are inference engines; Ollama and LM Studio package engines into local tools.

What is the difference between vLLM and Ollama?

vLLM is a GPU serving engine built for many concurrent users, using PagedAttention and continuous batching to raise throughput on data center hardware. Ollama is a local tool that downloads and runs quantized models on a single machine through a CLI, a desktop app, and a local API. vLLM suits shared production endpoints, and Ollama suits development and personal use.

vLLM vs SGLang: which is better?

Neither engine is better for every workload. Both are Apache-2.0 GPU serving engines with OpenAI-compatible servers and broad model support. SGLang's RadixAttention reuses cached prefixes across requests, which helps agent and multi-turn workloads with shared context, while vLLM supports a wide range of hardware backends. Benchmark both on your model, hardware, and traffic before choosing.

llama.cpp vs Ollama: what is the difference?

llama.cpp is a C/C++ library and server that runs GGUF models on CPUs, Apple Silicon, and GPUs, and it exposes many low-level options. Ollama is a separate tool that wraps model download, management, and serving behind simpler commands and a local API. Developers who want fine-grained control often use llama.cpp directly, while those who want a quick setup often start with Ollama.

TensorRT-LLM vs vLLM: what is the difference?

TensorRT-LLM is NVIDIA's open-source library for optimizing inference on NVIDIA GPUs, with custom attention kernels and FP8 and FP4 quantization tuned for that hardware. vLLM is a community project that runs on NVIDIA, AMD, TPU, and other accelerators and adds new models quickly. TensorRT-LLM suits NVIDIA-only fleets tuned for efficiency, and vLLM suits mixed hardware and fast model adoption.

Is Hugging Face TGI still maintained?

Text Generation Inference is in maintenance mode, and its GitHub repository is archived, so it no longer receives new features. Hugging Face's documentation recommends vLLM or SGLang for GPU serving and local engines such as llama.cpp or MLX for running models on personal hardware. Existing TGI deployments keep working, but new projects are better served by an actively developed engine.

Other categories

Product details change quickly; check each vendor’s site for current features, licensing and pricing.