vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use

Comparison · Inference

vLLM and Ollama are both open-source tools for running large language models, but they solve different problems: vLLM is a serving engine that keeps GPUs busy across many concurrent requests, and Ollama is a local runner that gets one model answering on one machine with a single command. The vLLM vs Ollama question usually stands in for a broader choice among six projects that engineers evaluate together: vLLM, Ollama, llama.cpp, SGLang, LM Studio, and TensorRT-LLM. This comparison checks each project's current scope against its official documentation and repository, then maps workloads to engines.

What are LLM inference engines?#

An LLM inference engine is software that loads a model's weights, schedules requests, manages KV cache memory, and runs the forward passes that generate tokens. The six projects here sit at different layers of that stack, which explains most of their differences in setup effort, concurrency, and hardware support.

Three layers are worth separating:

  • Runtimes and kernel libraries execute the model math on specific hardware: ggml (the library under llama.cpp), Apple's MLX, and PyTorch with custom CUDA or ROCm kernels.
  • Serving engines add batching, KV cache management, and an HTTP server: vLLM, SGLang, TensorRT-LLM, and llama.cpp's llama-server.
  • Applications add model downloads, a model library, lifecycle management, and a CLI or GUI on top of a runtime: Ollama and LM Studio.
Four layers from top to bottom: applications such as Ollama and LM Studio, serving engines such as vLLM, runtimes such as ggml and MLX, and hardware. ApplicationsModel library, CLI or GUIOllama: CLI, app, REST APILM Studio: GUI, lms CLIServing enginesBatching, KV cache, HTTP servervLLMSGLangTensorRT-LLMllama-serverRuntimes and kernelsModel math on specific hardwareggmlMLXPyTorch + CUDA or ROCmHardwareCPUsApple siliconConsumer GPUsData center GPUs
Figure 1: Ollama and LM Studio wrap lower-level runtimes; vLLM, SGLang, and TensorRT-LLM are serving engines.

The techniques these engines implement, from PagedAttention to speculative decoding, are covered in the LLM inference optimization guide. The serving engines directory lists more options in this category.

vLLM vs Ollama at a glance#

The table below summarizes each project's maintainer, license, design target, model format, and hardware coverage, based on each project's README and documentation as of November 2026. The design target column matters most: engines built for multi-user GPU serving and engines built for single-machine convenience make opposite trade-offs on memory allocation, setup, and defaults.

Engine Maintainer License Built for Model format Hardware
vLLM vLLM project, started at UC Berkeley's Sky Computing Lab Apache 2.0 High-throughput, multi-user GPU serving Hugging Face models; GGUF and pre-quantized formats NVIDIA, AMD, and Intel GPUs; x86, ARM, and PowerPC CPUs; TPU, Gaudi, Ascend, and Apple silicon through plugins
SGLang LMSYS, a non-profit Apache 2.0 Large-scale serving, agentic workloads, RL rollouts Hugging Face models NVIDIA, AMD Instinct, Google TPU, Intel, Huawei Ascend, Apple silicon, and others
TensorRT-LLM NVIDIA Apache 2.0 Serving on NVIDIA GPUs Hugging Face checkpoints, loaded directly NVIDIA GPUs only
llama.cpp ggml-org; the ggml.ai team joined Hugging Face in February 2026 MIT Portable inference with minimal setup GGUF CPUs (x86, ARM, RISC-V), Metal, CUDA, HIP, Vulkan, SYCL, and more
Ollama Ollama MIT Running models on one machine through a CLI, app, and API Ollama library models; imports GGUF and Safetensors macOS, Windows, Linux, Docker; NVIDIA, AMD (ROCm), Apple Metal, Vulkan
LM Studio LM Studio GUI app closed source, free for home and work; CLI, SDK, and MLX engine MIT Desktop app for local models GGUF (llama.cpp engine) and MLX macOS on Apple silicon, Windows x64 and ARM64, Linux x64

Is vLLM faster than Ollama?#

For many concurrent requests on a data center GPU, vLLM is designed to deliver higher total throughput, because continuous batching and PagedAttention pack many sequences into each forward pass. For one user on a laptop, the comparison rarely applies: Ollama runs natively on macOS, Windows, and Linux with modest GPUs, while vLLM's primary platform is a Linux GPU server.

The difference comes from opposite design assumptions.

vLLM assumes a dedicated accelerator. It preallocates a pool of KV cache blocks at startup, then schedules requests at the level of individual decode steps so the GPU stays saturated. Its feature list covers PagedAttention, continuous batching, chunked prefill, prefix caching, speculative decoding (n-gram, suffix, EAGLE, and others), and tensor, pipeline, data, expert, and context parallelism, plus disaggregated prefill and decode. vLLM serves an OpenAI-compatible API, an Anthropic Messages API, and gRPC, and its README claims support for more than 200 model architectures from Hugging Face. vLLM's installation docs list Linux as the operating system and state that Windows is not supported natively.

Ollama assumes a shared personal machine. Ollama loads models on demand when a request arrives, keeps them in memory for 5 minutes after last use by default, and unloads them to make room for other models. Concurrency is explicit: OLLAMA_NUM_PARALLEL sets how many requests each loaded model processes at once, required memory scales with that value times the context length, and OLLAMA_MAX_QUEUE caps waiting requests before the server returns 503 errors. Ollama's default context length depends on available VRAM, so set OLLAMA_CONTEXT_LENGTH or num_ctx explicitly for agents and long documents.

Ollama's engine has changed over time. In May 2025, Ollama introduced its own engine for multimodal models, built on the GGML tensor library, which it calls directly from Go; the README lists llama.cpp as a supported backend. On March 30, 2026, Ollama announced a preview of an MLX-based engine for Apple silicon in version 0.19. Ollama exposes a native REST API on port 11434 plus OpenAI-compatible endpoints (chat completions, completions, embeddings, models, and a stateless Responses endpoint) and an Anthropic-compatible /v1/messages endpoint.

Published vLLM vs Ollama benchmarks are hard to compare with each other, because they often mix quantization levels, GPUs, context lengths, and concurrency settings. A fair test fixes the model weights and precision, then sweeps concurrency from 1 to the expected peak while recording time to first token, inter-token latency, and throughput.

Ollama vs llama.cpp#

llama.cpp is a C and C++ inference library and server; Ollama is an application that packages a GGML-based engine with model management, a model library, and installers. Choose llama.cpp for direct control over build options, backends, and server flags. Choose Ollama for a managed experience where one command downloads a model, places it on available GPUs, and serves it.

llama.cpp's stated goal is LLM and vision-language model inference with minimal setup and high performance on a wide range of hardware. The project is MIT licensed, uses the GGUF file format, and supports integer quantization from 1.5 to 8 bits. Its backend list spans servers, desktops, and mobile chips: Metal, CUDA, HIP, Vulkan, SYCL, CANN, MUSA, OpenCL, Hexagon, WebGPU, and others, plus CPU paths for x86, ARM, and RISC-V.

llama-server, the bundled HTTP server, provides:

  • OpenAI-compatible chat completion, responses, and embedding routes, plus an Anthropic Messages-compatible route
  • Parallel requests for multiple users with continuous batching
  • Speculative decoding with a draft model
  • JSON schema-constrained output and function calling
  • Reuse of the previous request's KV cache through cache_prompt
  • Multi-GPU splitting by layer, row, or tensor through --split-mode
  • A built-in web UI and monitoring endpoints

On February 20, 2026, Georgi Gerganov and the ggml.ai team joined Hugging Face. The Hugging Face announcement states that llama.cpp remains fully open source and community driven, with the team keeping autonomy over technical direction.

Ollama adds a layer on top: a library of tagged models pulled by name, a Modelfile format for system prompts and parameters, automatic memory placement across GPUs, model unloading on idle, and a desktop app. Ollama also supports speculative decoding through a draft model setting (draft_num_predict) and a quantized KV cache through OLLAMA_KV_CACHE_TYPE. The cost of the extra layer is control: Ollama exposes a curated set of options through its API and Modelfile rather than every llama.cpp flag.

Ollama vs LM Studio#

Ollama and LM Studio both run models locally on one computer and serve an OpenAI-compatible API. Ollama is CLI-first and open source under the MIT license. LM Studio is GUI-first: the desktop app is closed source but free for home and work use, while its lms CLI, Core SDK, and MLX engine are MIT licensed.

LM Studio runs GGUF models through a llama.cpp engine on every platform and MLX models on Apple silicon Macs. The app includes a chat interface, offline document chat (a local form of retrieval-augmented generation), and support for acting as a Model Context Protocol client. Its local server exposes OpenAI-compatible endpoints (chat completions, completions, responses, embeddings, and models), an Anthropic-compatible Messages endpoint, and LM Studio's own REST API for loading, unloading, and downloading models. llmster runs LM Studio without a GUI on servers or in CI.

Two recent LM Studio changes make older comparisons out of date. On July 8, 2025, LM Studio dropped the separate commercial license requirement and became free for work use. LM Studio 0.4.0, released January 28, 2026, added parallel requests with continuous batching on its llama.cpp engine, with a default of 4 concurrent predictions; the release notes said MLX support for the feature was still in progress.

Choose LM Studio when teammates prefer a GUI, when you want to browse and compare models side by side, or when MLX on a Mac is the priority. Choose Ollama for scripting, Docker deployments, CI jobs, and tools that already integrate with Ollama's API.

vLLM vs SGLang#

vLLM and SGLang target the same workload, production serving on GPU clusters, and their feature lists overlap heavily: continuous batching, paged KV memory, prefix caching, speculative decoding, multi-GPU parallelism, and prefill-decode disaggregation. The practical differences are prefix-reuse design, scheduling behavior, and model-specific performance, which shift from release to release.

SGLang is hosted by LMSYS, licensed under Apache 2.0, and describes itself as optimized for agentic workloads, large-scale serving, and reinforcement learning rollouts. The original SGLang paper (Zheng et al.) introduced two runtime ideas: RadixAttention, which stores KV cache prefixes in a radix tree with least-recently-used eviction and cache-aware scheduling, and compressed finite state machines that speed up constrained decoding for structured outputs. The authors reported up to 6.4 times higher throughput than the systems they compared against on workloads including agent control, JSON decoding, and multi-turn chat.

Current SGLang documentation recommends EAGLE-3 for speculative decoding, with EAGLE-2, multi-token prediction, and n-gram drafting also available, and supports prefill-decode disaggregation with Mooncake and NIXL as KV transfer backends. SGLang's hardware list is broad: NVIDIA data center and RTX GPUs, AMD Instinct, Google TPU, Intel GPUs and Xeon CPUs, Huawei Ascend NPUs, and Apple silicon. SGLang also ships SGLang Diffusion for image and video generation models.

vLLM's prefix caching takes a different approach to the same goal: vLLM hashes each full KV block together with its parent block's hash, so a block is reused only when the whole preceding prefix matches. Both designs reward prompts that keep stable content first.

Because both engines serve OpenAI-compatible APIs, a head-to-head test costs little. Run one load generator against both with the same model, precision, and parallelism settings, and compare goodput at your latency target rather than peak throughput.

TensorRT-LLM vs vLLM#

TensorRT-LLM is NVIDIA's open-source inference library and runs only on NVIDIA GPUs, while vLLM runs on NVIDIA, AMD, and Intel GPUs and other accelerators through plugins. Teams choose TensorRT-LLM for NVIDIA-maintained kernels and integration with NVIDIA Dynamo and Triton Inference Server, and choose vLLM for hardware portability and broad model coverage.

TensorRT-LLM is licensed under Apache 2.0 and is now built on PyTorch with a high-level Python LLM API. Its documented features include in-flight batching, a paged KV cache with block reuse, chunked prefill, speculative decoding (EAGLE, multi-token prediction, and n-gram), disaggregated serving in beta, wide expert parallelism for mixture-of-experts models, FP8 on H100 and later GPUs, and FP4 on Blackwell B200. trtllm-serve starts an OpenAI-compatible server with completions, chat completions, and models endpoints.

The largest recent change: TensorRT-LLM's migration guide on the TensorRT backend removal states that the TensorRT engine backend has been removed and that PyTorch is now the sole execution backend. Hugging Face checkpoints load directly, with no separate checkpoint conversion or engine build step. Comparisons that describe a mandatory compile step for TensorRT-LLM are out of date. The README also notes that anonymous usage telemetry is enabled by default and can be disabled with an environment variable, a file, an API setting, or a CLI flag.

Feature comparison#

The table below compares the six engines on the features that most often settle a deployment choice: how each handles concurrent requests, which speculative decoding methods each documents, how each spreads a model across GPUs, and which HTTP APIs each serves. Entries reflect each project's official documentation as of November 2026.

Engine Concurrent requests Speculative decoding Multi-GPU HTTP APIs
vLLM Continuous batching with chunked prefill n-gram, suffix, EAGLE, MTP, draft models Tensor, pipeline, data, expert, and context parallelism; disaggregated prefill and decode OpenAI-compatible, Anthropic Messages, gRPC
SGLang Continuous batching with RadixAttention prefix reuse EAGLE-2, EAGLE-3, MTP, n-gram Multi-GPU parallelism; prefill-decode disaggregation OpenAI-compatible
TensorRT-LLM In-flight batching with chunked prefill EAGLE, MTP, n-gram Tensor, pipeline, and expert parallelism across nodes; disaggregated serving (beta) OpenAI-compatible via trtllm-serve
llama.cpp Continuous batching across parallel server slots Draft model Split by layer, row, or tensor across GPUs OpenAI-compatible, Anthropic Messages
Ollama Per-model parallel requests plus a request queue Draft models via draft_num_predict Spreads a model across GPUs when it does not fit on one Native REST, OpenAI-compatible, Anthropic-compatible
LM Studio Parallel requests with continuous batching (llama.cpp engine, 0.4.0 and later) Draft model Per-model GPU offload settings OpenAI-compatible, Anthropic-compatible, LM Studio REST

The shared OpenAI-compatible surface means switching engines rarely requires application changes. The same client code reaches any of them:

from openai import OpenAI
 
# Default local endpoints; only base_url and model change between engines.
ENDPOINTS = {
    "vllm": "http://localhost:8000/v1",
    "sglang": "http://localhost:30000/v1",
    "ollama": "http://localhost:11434/v1",
    "lm_studio": "http://localhost:1234/v1",
}
 
client = OpenAI(base_url=ENDPOINTS["ollama"], api_key="unused-locally")
reply = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Define PagedAttention in one sentence."}],
)
print(reply.choices[0].message.content)

Compatibility covers request and response shapes, not behavior. Tool calling formats, structured output enforcement, and reasoning-token handling differ by engine and model, so test those paths explicitly. If several engines or providers sit behind one application, an AI gateway can route between them and record usage in one place. Bifrost, for example, lists vLLM, Ollama, and SGLang as providers alongside hosted model APIs.

Which should you use?#

Pick the engine by where it runs and how many requests it serves at once. For a laptop or workstation, start with Ollama, LM Studio, or llama.cpp. For a multi-user API on Linux GPU servers, start with vLLM or SGLang. For an NVIDIA-only fleet, evaluate TensorRT-LLM. Then benchmark on your own model and traffic.

Scenario Start with Why
Prototyping on a Mac or Windows laptop Ollama or LM Studio One-step install, model library, local API; MLX support on Apple silicon
Teammates who prefer a GUI or want side-by-side model comparison LM Studio Desktop chat, model browser, split-view chats
Scripts, Docker, and CI on a single machine Ollama CLI-first, official Docker image, native and OpenAI-compatible APIs
CPU-only, edge, or unusual hardware llama.cpp Backends from CPUs to mobile chips, dependency-free C/C++ build, GGUF quantization down to 1.5 bits
Public or multi-tenant API on Linux GPU servers vLLM Throughput-oriented scheduling, broad model and accelerator coverage
Agents and multi-turn traffic with heavy prefix sharing, or RL rollouts SGLang, benchmarked against vLLM RadixAttention prefix reuse and an agentic-workload focus
NVIDIA-only fleet using Dynamo or Triton TensorRT-LLM NVIDIA-maintained kernels and integrations, FP8 and FP4 paths
No interest in operating GPUs A hosted inference provider Per-token pricing with no GPU operations

Hosted options are listed in the inference providers directory. One workable pattern runs two engines: Ollama or LM Studio on developer laptops and vLLM or SGLang in production, with the same OpenAI-compatible client in both places. Production deployments also need tracing and token accounting, which the LLM observability guide covers.

For the mechanics behind these engines, read the LLM inference optimization guide; for picking which model to serve, see LLM benchmarks and the inference topic hub.

Frequently asked questions

Is vLLM faster than Ollama?

vLLM is built for higher total throughput when many requests arrive at once on a data center GPU, because continuous batching and PagedAttention pack many sequences into each forward pass. Ollama processes a configurable number of parallel requests per model and queues the rest. For a single user on a laptop, Ollama is usually the practical option, since vLLM targets Linux GPU servers. Benchmark both on your own hardware and traffic.

Can vLLM run on Mac?

vLLM can run on Apple silicon Macs in two ways: the CPU backend, or vLLM-Metal, a community-maintained hardware plugin under the vllm-project GitHub organization that uses MLX as its compute backend and requires macOS 15 or later. vLLM's primary platform is Linux, and its documentation states that Windows is not supported natively; Windows users run it under WSL. For everyday local use on a Mac, Ollama, LM Studio, and llama.cpp are simpler.

Is Ollama good for production?

Ollama works for production cases with modest concurrency on one machine, such as internal tools, offline or air-gapped assistants, and edge deployments. Ollama processes a configurable number of parallel requests per model, memory grows with that number times the context length, and excess requests wait in a queue. For public APIs serving many concurrent users across GPUs, engines built for throughput, such as vLLM, SGLang, or TensorRT-LLM, fit better.

What is the difference between Ollama and llama.cpp?

llama.cpp is an open-source C and C++ inference library and server that runs GGUF models on CPUs and many GPU backends, configured through command-line flags. Ollama is an application that packages a GGML-based engine with a model library, a Modelfile format, automatic GPU memory placement, model loading and unloading, and installers or install scripts for macOS, Windows, and Linux. Ollama trades low-level control for convenience.

Is Ollama open source?

Yes. Ollama is open source under the MIT license, and its code is on GitHub. vLLM, SGLang, and TensorRT-LLM use the Apache 2.0 license, and llama.cpp uses MIT. LM Studio is the exception in this group: its desktop GUI app is closed source but free for home and work use, while its lms command-line tool, Core SDK, and MLX inference engine are MIT licensed.

Which is better, vLLM or SGLang?

Neither is better across every workload. vLLM and SGLang both provide continuous batching, paged KV memory, prefix caching, speculative decoding, multi-GPU parallelism, and prefill-decode disaggregation. SGLang's RadixAttention targets workloads with heavy prefix sharing, such as multi-turn agents. Relative performance depends on the model, hardware, engine version, and traffic shape, so run the same load test against both before committing.

Terms in this guide

Cite this guide

The AI Engineering Editors. "vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/vllm-vs-ollama

Title
vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use
Author
The AI Engineering Editors
Publisher
The AI Engineering
Published
Updated
View as Markdown

The AI Engineering Editors ·