---
title: "How to self-host an LLM: hardware, serving engines, and a production setup"
description: "Self-hosted LLM guide: size GPU memory for weights and KV cache, run models locally with Ollama or llama.cpp, and serve them with vLLM, SGLang, or Kubernetes."
summary: "Self-hosting an LLM means running an open-weights model on hardware you control, from a laptop to a Kubernetes cluster. This guide covers the memory arithmetic, the commands for each deployment tier, and what production adds around the engine: auth, a gateway, metrics, autoscaling, and evals."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/self-hosted-llm"
markdown_url: "https://theaiengineering.com/articles/self-hosted-llm.md"
topic: "LLM inference and serving"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-11-18"
updated: "2026-11-18"
keywords: ["self hosted llm","run llm locally","local llm","self-hosting llm","self host llm"]
citation: "The AI Engineering Editors. \"How to self-host an LLM: hardware, serving engines, and a production setup.\" The AI Engineering, updated Nov 18, 2026. https://theaiengineering.com/articles/self-hosted-llm"
---

# How to self-host an LLM: hardware, serving engines, and a production setup

## TL;DR

- A self-hosted LLM is an open-weights model served by an inference engine on infrastructure your team operates, usually behind an OpenAI-compatible API.
- GPU memory must hold the weights (parameters times bytes per parameter), the KV cache (which grows with context length and concurrent sequences), and runtime overhead.
- Qwen3-32B needs 65.6 GB for BF16 weights or 19.8 GB as a Q4_K_M GGUF file, and its KV cache takes 256 KiB per token at 16-bit precision, so 16 concurrent 8,192-token sequences need another 34.4 GB.
- Ollama, LM Studio, and llama.cpp fit one machine; vLLM or SGLang in Docker fit a shared GPU server; KServe, llm-d, NVIDIA Dynamo, and the Gateway API Inference Extension add routing and autoscaling on Kubernetes.
- Self-hosting pays off at sustained, predictable volume, so compare cost per million tokens at your real GPU utilization against API prices for the same traffic before committing.

A self-hosted LLM is a large language model that runs on hardware your team controls, from a developer laptop to a GPU cluster, instead of behind a vendor's API. Self-hosting an LLM takes five decisions: which model and license, how much memory it needs, which engine serves it, where that engine runs, and what sits in front of it in production. This guide covers each one with commands checked against each project's documentation as of November 2026. For a side-by-side of engines, read the [vLLM vs Ollama comparison](https://theaiengineering.com/articles/vllm-vs-ollama); for throughput and latency tuning, read the [LLM inference optimization guide](https://theaiengineering.com/articles/llm-inference-optimization). Both belong to the [LLM inference hub](https://theaiengineering.com/topics/llm-inference).

## What is a self-hosted LLM?

A self-hosted LLM is an open-weights model served by an inference engine on infrastructure your team operates, exposed to applications through an HTTP API.

The model is an [open-weights model](https://theaiengineering.com/glossary/open-weights-model): a lab published its trained parameters for download under a license. The [inference engine](https://theaiengineering.com/glossary/inference-engine) loads those weights, batches requests, manages GPU memory, and streams tokens. The API is almost always an [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api), so application code written for a hosted provider moves to your own server by changing a base URL, a key, and a model name.

A local LLM is the single-machine case: one person runs a model on their own laptop or workstation. Self-hosting also covers shared GPU servers and Kubernetes clusters that serve many users, which is where most of the operational work lives.

**Figure 1: The self-hosted LLM stack. A hosted API hides the bottom four layers; self-hosting makes them your team's job.**

Six layers from application at the top through AI gateway, OpenAI-compatible API, inference engine, and model weights down to hardware, with the bottom four marked as what self-hosting adds.

Layers, top to bottom:

1. **Application**: Chat UI, agents, batch jobs, RAG pipelines
2. **AI gateway**: Per-team keys, rate limits, fallbacks, usage logs
3. **OpenAI-compatible API**: /v1/chat/completions, /v1/embeddings, /v1/models
4. **Inference engine**: Batching, KV cache, token streaming (vLLM, SGLang, Ollama, llama.cpp)
5. **Model weights**: Open weights at a chosen precision (BF16, FP8, 4-bit GGUF or AWQ)
6. **Hardware** (Laptop, GPU server, Kubernetes GPU nodes)

You run these spans OpenAI-compatible API through Hardware.

## Why self-host an LLM, and when not to

Teams self-host LLMs for four reasons: control over where prompts and outputs go, the ability to run offline or in air-gapped networks, lower cost at sustained high volume, and freedom to customize and pin model versions. Self-hosting fits poorly when traffic is low or spiky, when no one can operate GPUs, or when a task needs a proprietary frontier model.

The reasons in more detail:

- **Data control.** Prompts, retrieved documents, and outputs stay on networks you control, which simplifies data residency and vendor reviews. Audit outbound calls anyway: vLLM [collects anonymous usage data](https://docs.vllm.ai/en/latest/usage/usage_stats/) about hardware and configuration by default, and setting `VLLM_NO_USAGE_STATS=1` or `DO_NOT_TRACK=1` turns it off.
- **Offline and air-gapped use.** Once weights are on local disk, an engine needs no internet connection. Download on a connected machine with `hf download <repo_id>`, copy the cache across, and set `HF_HUB_OFFLINE=1` so Hugging Face libraries read only cached files. Ollama reads models from the directory named in `OLLAMA_MODELS`.
- **Cost at sustained volume.** GPUs bill by the hour whether they serve one request or thousands. A GPU kept busy can serve tokens below API prices; an idle one cannot.
- **Customization.** You can serve fine-tuned models, keep a model version fixed for as long as you need, and set decoding parameters per route. vLLM serves several fine-tuned LoRA adapters on one base model with `--enable-lora` and `--lora-modules name=path`, and clients pick an adapter by passing its name as the model.

The most capable proprietary models are offered only through their vendors' APIs and clouds, so if your evals show that no open-weights model handles the task, self-hosting cannot fix that. Low or bursty traffic leaves GPUs idle. And someone has to own drivers, capacity, upgrades, and on-call. A middle path exists: hosted [inference providers](https://theaiengineering.com/platforms/inference-providers) serve open-weights models per token, so you can test a model on real traffic before buying or renting GPUs.

| Factor | Favors self-hosting | Favors a hosted API |
|---|---|---|
| Traffic | Sustained, predictable volume | Low, bursty, or unknown volume |
| Data | Must stay on your network or run offline | Provider terms meet your requirements |
| Model | An open-weights model passes your evals | Only a proprietary model passes |
| Versioning | You need a pinned model that never changes underneath you | Provider-managed updates are acceptable |
| Team | Someone owns GPUs, drivers, and on-call | No infrastructure capacity |

### How to compare self-hosting cost with an API

Compare cost per million tokens on both sides, using your own measurements rather than published peak throughput. The self-hosted side is a ratio of hourly cost to tokens actually served:

```text
Self-hosted cost per 1M tokens =
    (GPU price per hour × GPU count + other infrastructure per hour)
  ÷ (tokens served per hour at your latency target)
  × 1,000,000

Tokens served per hour = sustained throughput (tokens/s) × 3,600 × utilization
```

The result depends on throughput, utilization, and fixed costs:

1. **Sustained throughput.** Load-test the engine with your real prompt and output lengths, and record the highest throughput that still meets your p95 latency target. Long prompts consume prefill capacity, so a chat benchmark says little about a retrieval workload.
2. **Utilization.** Utilization is the share of billed GPU hours that carry traffic. A GPU billed for 24 hours that serves traffic for 6 of them runs at 25% utilization, which makes each token four times as expensive as the full-load figure.
3. **Everything else.** Add redundancy (two replicas for availability means each carries half the load), storage, networking, and the engineering time to operate the stack.

On the API side, multiply each provider's input and output prices by your input and output token counts, and include cached-input discounts if your prompts repeat. Hourly rental options are listed in the [GPU clouds directory](https://theaiengineering.com/platforms/gpu-clouds); check current prices on each provider's own pricing page.

## How to choose a model to self-host

Choose a self-hosted model by license first, then by the smallest size that passes your own evaluation set at the precision you plan to deploy. Open-weights licenses range from permissive licenses such as Apache 2.0 and MIT to custom community licenses with usage thresholds, attribution rules, acceptable use policies, or non-commercial terms.

License terms differ in ways that matter for products:

- **Permissive licenses.** Apache 2.0 and MIT allow commercial use, modification, and redistribution with notice requirements. Qwen3 and OpenAI's gpt-oss models are released under Apache 2.0.
- **Custom community licenses.** The [Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) requires a separate license from Meta if your products had more than 700 million monthly active users in the month before the model's release date, requires "Built with Llama" to be displayed when you distribute the model or products built with it, and incorporates an acceptable use policy.
- **Non-commercial licenses.** Some releases ship under terms such as CC-BY-NC-4.0, which do not permit commercial use.

The [release tracker](https://theaiengineering.com/releases) records licenses, parameter counts, and context windows for open-weights releases, each linked to the lab's announcement, which makes it a practical source for a shortlist.

Size is a trade-off between quality, memory, and speed. Larger models tend to score higher on general benchmarks, but each parameter costs memory and memory bandwidth. A [small language model](https://theaiengineering.com/glossary/small-language-model) often handles narrow tasks such as classification, extraction, or routing at a fraction of the cost. Mixture-of-experts models split the difference: total parameters set the memory footprint, while active parameters set the compute per token. OpenAI's [gpt-oss model card](https://huggingface.co/openai/gpt-oss-20b) lists gpt-oss-120b at 117B total and 5.1B active parameters and states that it fits on a single 80 GB GPU, because its MoE weights use MXFP4 quantization.

Shortlist two or three candidates, run your task-specific evaluation set on each at the precision you will deploy, and only then size hardware.

## How much GPU memory does a self-hosted LLM need?

GPU memory must hold three things: the model weights, the KV cache for every token in flight, and runtime overhead. Weights equal parameters times bytes per parameter at the chosen precision. The KV cache grows with context length and with the number of concurrent sequences, and it often decides how many users one GPU can serve.

```text
Weights   = parameters × bytes per parameter
            (BF16 or FP16 = 2, FP8 or INT8 = 1, 4-bit = 0.5 plus scales)
KV cache  = 2 × layers × KV heads × head dim × bytes per value
            × tokens per sequence × concurrent sequences
Total     ≈ weights + KV cache + overhead (activations, CUDA graphs, runtime)
```

[Quantization](https://theaiengineering.com/glossary/quantization) shrinks the first term by storing weights in fewer bits. The [KV cache](https://theaiengineering.com/glossary/kv-cache) term uses four numbers from the model's `config.json`: `num_hidden_layers`, `num_key_value_heads`, `head_dim`, and the cache data type. The factor of 2 covers keys and values.

### Worked example: Qwen3-32B at BF16 and 4-bit

[Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) is a dense model released under Apache 2.0. Its model card and `config.json` list 32.8B parameters, 64 layers, 8 key-value heads shared by 64 query heads, a head dimension of 128, BF16 weights, and a native context length of 32,768 tokens.

**Weights.** At BF16, 32.8 billion parameters × 2 bytes = 65.6 GB, which matches the 66 GB Ollama lists for its `qwen3:32b-fp16` build. At 4 bits, the theoretical size is 32.8 billion × 0.5 bytes = 16.4 GB. Real 4-bit files are larger: Qwen's own Q4_K_M [GGUF](https://theaiengineering.com/glossary/gguf) file is 19.8 GB, about 4.8 bits per weight, because k-quant blocks store scales alongside the weights (Hugging Face's [GGUF documentation](https://huggingface.co/docs/hub/gguf) puts Q4_K at 4.5 bits per weight) and the file mixes quantization types across tensors.

**KV cache per token.** At 16-bit precision, 2 × 64 layers × 8 KV heads × 128 × 2 bytes = 262,144 bytes, or 256 KiB per token. One full 32,768-token sequence needs 8.6 GB. Sixteen concurrent sequences of 8,192 tokens each hold 131,072 tokens, or 34.4 GB.

| Configuration | Weights | KV cache | Total before overhead | Fits on |
|---|---|---|---|---|
| BF16 weights, BF16 cache, 16 × 8,192 tokens | 65.6 GB | 34.4 GB | 100.0 GB | Two 80 GB GPUs with tensor parallelism |
| FP8 weights, FP8 cache, 16 × 8,192 tokens | 32.8 GB | 17.2 GB | 50.0 GB | One 80 GB GPU |
| 4-bit Q4_K_M weights, BF16 cache, 16 × 8,192 tokens | 19.8 GB | 34.4 GB | 54.2 GB | One 80 GB GPU |
| 4-bit Q4_K_M weights, BF16 cache, 1 × 8,192 tokens | 19.8 GB | 2.1 GB | 21.9 GB | A 24 GB GPU, with little headroom |

The same arithmetic generalizes to other models:

- **Concurrency costs as much as the model.** In the 4-bit row, the cache for 16 users is larger than the weights. Quantizing weights does nothing for the cache; quantizing the cache does. vLLM's `--kv-cache-dtype fp8` and SGLang's `--kv-cache-dtype fp8_e4m3` halve it relative to 16-bit.
- **Context limits bound the worst case.** Setting `--max-model-len` in vLLM or `--context-length` in SGLang caps the cache any single request can claim.
- **Splitting across GPUs is a sizing tool.** The BF16 row needs [tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism) across two GPUs. FP8 computation in vLLM requires NVIDIA GPUs with compute capability 8.9 or higher (Ada Lovelace, Hopper, Blackwell); older GPUs from Turing onward run FP8 checkpoints as weight-only W8A16.
- **Precision changes quality.** Qwen's [Qwen3-32B-AWQ model card](https://huggingface.co/Qwen/Qwen3-32B-AWQ) reports 79.4 on AIME24 in thinking mode for the 4-bit model against 81.4 for BF16, while GPQA moved from 68.4 to 69.0. Because scores moved in both directions, the quantized model needs its own eval run on your task.

Treat the formula as an upper bound for unusual architectures: models with sliding-window, hybrid, or latent attention store less cache per token than standard grouped-query attention. Engines report what they actually allocated. vLLM claims a configurable share of GPU memory (`--gpu-memory-utilization`), loads the weights, profiles activation memory, gives the rest to the cache in fixed-size blocks through [PagedAttention](https://theaiengineering.com/glossary/paged-attention), and logs the resulting capacity at startup. When the cache fills, vLLM preempts requests and recomputes them once space frees up, which shows up as latency spikes.

## How to run an LLM locally on a laptop or workstation

To run an LLM locally, install Ollama, LM Studio, or llama.cpp, download a 4-bit model that fits your RAM or VRAM, and call the local OpenAI-compatible endpoint. These tools target one machine with one or a few users, run on macOS, Windows, and Linux, and run quantized models on Apple silicon and consumer GPUs; llama.cpp and Ollama also run on CPU-only machines.

**Ollama** is the shortest path from nothing to a working local LLM:

```bash
# Linux install script; macOS and Windows use the desktop installer
curl -fsSL https://ollama.com/install.sh | sh

# Download and chat with a model (qwen3:8b is a 5.2 GB 4-bit build)
ollama run qwen3:8b
```

Ollama serves a native REST API on port 11434 and an OpenAI-compatible API at `http://localhost:11434/v1`. A few settings matter when local use turns into shared use. According to the [Ollama FAQ](https://docs.ollama.com/faq), Ollama binds to 127.0.0.1:11434 by default, `OLLAMA_HOST` changes the bind address, and `OLLAMA_NUM_PARALLEL` sets how many requests each model processes at once (default 1). Ollama's [context length documentation](https://docs.ollama.com/context-length) sets the default context from available VRAM (4k tokens below 24 GiB), so set `OLLAMA_CONTEXT_LENGTH` explicitly for agents and long documents. On Linux, Ollama runs as a systemd service, and environment variables go in an override created with `sudo systemctl edit ollama`.

**LM Studio** pairs a desktop app with the `lms` CLI. For servers and other machines without a GUI, LM Studio packages its core as llmster:

```bash
curl -fsSL https://lmstudio.ai/install.sh | bash   # installs llmster
lms daemon up
lms get llama-3.1-8b@q4_k_m
lms load <model_key> --context-length 4096
lms server start
```

LM Studio's documentation examples use port 1234 (`http://localhost:1234/v1`). Its API does not require authentication by default; API tokens can be turned on in Server Settings.

**llama.cpp** is the C and C++ runtime underneath much of the local ecosystem, and it gives the most direct control over flags:

```bash
# Install the llama command-line tools, then serve a model from Hugging Face
curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

# Or run the llama-server binary with explicit limits and a key
llama-server -hf ggml-org/Qwen3.5-0.8B-GGUF -c 8192 --api-key "$LLAMA_API_KEY"
```

According to the [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), `-hf` downloads a GGUF file from a Hugging Face repository and defaults to the Q4_K_M quantization when you do not name one. The server listens on 127.0.0.1:9931 by default, `-np` sets the number of parallel slots, `--metrics` turns on a Prometheus endpoint, and `GET /health` returns 503 while the model loads and 200 once it is ready.

## How to serve a self-hosted LLM from a single GPU server

A single Linux GPU server is the next tier: run vLLM or SGLang in Docker, point it at a model on Hugging Face or local disk, and expose an OpenAI-compatible endpoint to services on your private network. Both engines use [continuous batching](https://theaiengineering.com/glossary/continuous-batching) and block-based KV memory, so one GPU serves many concurrent requests instead of one at a time.

The FP8 row of the worked example fits one 80 GB GPU. This command follows the [vLLM Docker deployment guide](https://docs.vllm.ai/en/latest/deployment/docker.html):

```bash
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_API_KEY=$VLLM_API_KEY" \
  --env "VLLM_NO_USAGE_STATS=1" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-32B-FP8 \
  --served-model-name qwen3-32b \
  --max-model-len 32768
```

`--ipc=host` gives the container the host's shared memory, which PyTorch uses between processes, particularly for tensor-parallel inference. `HF_TOKEN` is needed only for gated models. Once the server logs that it is ready, call it like any OpenAI endpoint:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-32b", "messages": [{"role": "user", "content": "Reply with one word."}]}'
```

The flags that map to the sizing arithmetic:

| vLLM flag | What it controls | Why it matters |
|---|---|---|
| `--max-model-len` | Maximum context per request (prompt plus output) | Caps the cache one request can claim |
| `--gpu-memory-utilization` | Fraction of GPU memory vLLM may use | Memory left after weights becomes KV cache |
| `--max-num-seqs` | Maximum sequences processed per iteration | Bounds concurrency and cache pressure |
| `--kv-cache-dtype fp8` | Data type of cached keys and values | Halves cache size relative to 16-bit |
| `--tensor-parallel-size` | Number of GPUs one replica spans | Needed for the BF16 row of the worked example |
| `--served-model-name` | Model name clients send in requests | Keeps a stable alias when the weights change |
| `--revision` | Hugging Face branch, tag, or commit | Pins the exact weights you evaluated |

SGLang serves the same model with equivalent options (`--context-length`, `--mem-fraction-static`, `--tp`, `--kv-cache-dtype`). Its server binds to 127.0.0.1:30000 by default, so set `--host` inside a container:

```bash
docker run --gpus all --shm-size 32g --ipc=host \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8 \
    --host 0.0.0.0 --port 30000 --api-key "$SGLANG_API_KEY" --enable-metrics
```

Before other teams depend on the server, pin the image to a specific version tag instead of `latest`, add a Docker restart policy, put a TLS-terminating proxy in front, and point your load balancer's health check at vLLM's `/health` endpoint. Choosing between vLLM and SGLang for a given model is a benchmarking question: run the same load test against both with the same weights and precision.

## How to self-host LLMs on Kubernetes

On Kubernetes, model servers still run vLLM or SGLang in pods; the added layers handle model-aware routing, autoscaling, and rollouts. As of November 2026, the main open-source options are KServe, llm-d, NVIDIA Dynamo, and the Gateway API Inference Extension, which turns a standard Kubernetes gateway into a router that picks pods by model-server metrics.

| Project | What it adds | Engines | Status, November 2026 |
|---|---|---|---|
| [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) | The InferencePool resource and an endpoint picker that routes each request using model-server metrics such as cache use and queue length | Model servers in the pool | InferencePool v1 has been GA since v1.0.0; v1.6.0 (August 17, 2026) moved the endpoint picker into the llm-d project; latest release v1.6.2 (September 17, 2026) |
| [llm-d](https://github.com/llm-d/llm-d) | Prefix-cache and load-aware routing, tiered KV cache offload, prefill-decode disaggregation, and wide expert parallelism, packaged as benchmarked recipes with Helm charts | vLLM, SGLang | v1.0.0 released October 20, 2026; CNCF Sandbox project |
| [KServe](https://kserve.github.io/website/) | InferenceService for general model serving, plus LLMInferenceService, built on llm-d, with autoscaling and canary rollouts | vLLM, directly or through llm-d | v0.21.0 released September 25, 2026 |
| [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) | Multi-node orchestration: disaggregated prefill and decode, KV-aware routing, KV cache offload to CPU, SSD, and remote storage, and an SLA-driven autoscaler called Planner | vLLM, SGLang, TensorRT-LLM | v1.6.2 released November 5, 2026; Apache 2.0; deploys through the Dynamo platform and Helm charts |

The projects overlap more than the table suggests. The Inference Extension names llm-d's router as an example of a production endpoint picker, KServe's LLMInferenceService builds on llm-d, and llm-d runs vLLM or SGLang underneath. In practice, KServe fits teams that already use it for other models, llm-d fits vLLM or SGLang fleets that need cache-aware routing without KServe, and Dynamo fits large multi-node clusters running disaggregated serving. More options are listed in the [serving engines directory](https://theaiengineering.com/platforms/serving-engines).

KServe's LLMInferenceService shows how compact the declaration can be. This manifest follows the example in KServe's documentation, with the model swapped for the FP8 build sized above:

```yaml
apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: qwen3-32b
spec:
  model:
    uri: hf://Qwen/Qwen3-32B-FP8
    name: Qwen/Qwen3-32B-FP8
  replicas: 2
  template:
    containers:
      - name: main
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "1"
  router:
    gateway: {}     # managed Gateway API gateway
    route: {}       # managed HTTPRoute
    scheduler: {}   # managed endpoint picker
```

**Figure 2: On Kubernetes, the AI gateway owns keys, limits, and fallbacks, while the inference gateway and endpoint picker choose a model server pod for each request.**

An application calls an AI gateway, which routes to a Kubernetes inference gateway that picks a vLLM pod by cache and queue metrics, with a hosted API as fallback.

Components:

- **Application**: chat, agents, jobs
- **AI gateway**: keys, rate limits
- **Hosted API**: fallback provider
- **Endpoint picker**: cache, queue depth
- **Inference gateway**: InferencePool
- **Autoscaler**: KEDA, HPA, or WVA
- **vLLM pod**: replica 1
- **vLLM pod**: replica 2
- **vLLM pod**: replica 3

Groups:

- Kubernetes cluster: Endpoint picker, Inference gateway, Autoscaler, vLLM pod, vLLM pod, vLLM pod

Connections:

- Application → AI gateway
- AI gateway → Hosted API: fallback
- AI gateway → Inference gateway: primary
- Inference gateway → Endpoint picker: pick pod
- Inference gateway → vLLM pod
- vLLM pod → Endpoint picker: metrics
- Autoscaler → vLLM pod: add or remove

## Production concerns for a self-hosted LLM

A model answering on port 8000 is not yet a production service. Production self-hosting adds authentication and TLS, per-team rate limits, a fallback path for outages and saturation, metrics and traces, autoscaling with a plan for cold starts, and a release process that evaluates each model change before traffic moves to it.

### Authentication and network exposure

Inference engines ship with thin authentication, and some ship with none. Treat the engine port as an internal service and put identity, TLS, and per-caller limits in a layer in front of it.

| Engine | Default listen address | Built-in authentication |
|---|---|---|
| Ollama | 127.0.0.1:11434 | None; OpenAI clients must send a key value, which Ollama ignores |
| LM Studio | Port 1234 in its documentation examples | Optional API tokens, off by default |
| llama.cpp server | 127.0.0.1:9931 | One or more keys through `--api-key` or `LLAMA_API_KEY` |
| vLLM | Port 8000 in its documentation examples | One key through `--api-key` or `VLLM_API_KEY` for routes under `/v1`, `/v2`, `/inference`, and `/cohere` |
| SGLang | 127.0.0.1:30000 | One key through `--api-key` |

vLLM's [OpenAI-compatible server documentation](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/) states that endpoints outside those prefixes, notably `/invocations`, are not authenticated, and warns against relying on `--api-key` alone. A single shared key also carries no user identity, so it cannot attribute usage or revoke one team's access.

### Rate limits and fallbacks through an AI gateway

An [AI gateway](https://theaiengineering.com/articles/what-is-an-ai-gateway) in front of the engines issues a key per team or service, enforces rate limits and token budgets, logs usage, and routes to a hosted API when the self-hosted pool is down or saturated. [Bifrost](https://docs.getbifrost.ai/providers/supported-providers/overview), for example, lists vLLM, Ollama, and SGLang (as SGL) among its supported providers, and its [fallbacks](https://docs.getbifrost.ai/features/retries-and-fallbacks) setting sends a failed request to the next provider and model in a list. Decide per route which data may fall back to a hosted provider, because a fallback sends that request outside your network.

### Observability

vLLM exposes Prometheus metrics at `/metrics`, including `vllm:num_requests_running`, `vllm:num_requests_waiting`, `vllm:kv_cache_usage_perc`, `vllm:time_to_first_token_seconds`, `vllm:inter_token_latency_seconds`, `vllm:e2e_request_latency_seconds`, and `vllm:num_preemptions`, according to its [metrics documentation](https://docs.vllm.ai/en/latest/usage/metrics.html). SGLang publishes metrics with `--enable-metrics`, and llama.cpp with `--metrics`. Alert on queue depth, cache usage near 100%, rising preemptions, and p95 time to first token. Engine metrics describe the server; per-request traces, token accounting, and output quality live one layer up, which the [LLM observability guide](https://theaiengineering.com/articles/llm-observability) covers.

### Autoscaling and cold starts

Scale on inference signals, not CPU. KServe's Workload Variant Autoscaler computes replica counts from KV cache utilization, queue depth, and request saturation, then applies them through a Horizontal Pod Autoscaler or a KEDA ScaledObject. Dynamo's Planner scales prefill and decode pools against latency targets. On a custom stack, `vllm:num_requests_waiting` and `vllm:kv_cache_usage_perc` are workable scaling signals.

A new replica is slow to become useful. It waits for a GPU node, pulls a large image, fetches tens of gigabytes of weights, loads them into GPU memory, and compiles and captures CUDA graphs. vLLM's [optimization documentation](https://docs.vllm.ai/en/stable/configuration/optimization/) lists several ways to shorten startup:

- Bake the `torch.compile` cache from `VLLM_CACHE_ROOT` (default `~/.cache/vllm`) into the container image or copy it between machines. Any change to the model, configuration, torch build, or GPU model invalidates it.
- Pass the `--kv-cache-memory` value vLLM logged on a previous boot to skip memory profiling on the same GPU.
- Use `--enforce-eager`, which skips compilation and CUDA graph capture for the fastest startup at the cost of slower decoding.
- Stream Safetensors weights with `--load-format runai_streamer`, and keep weights on node-local or shared storage instead of downloading them per pod.

Keep a minimum replica count for latency-sensitive routes and reserve scale-to-zero for batch and internal workloads that tolerate a slow first request.

### Model updates and evaluation before swapping

Treat a model change like a schema migration. Pin weights by commit with `--revision`, pin engine image tags, and keep the `--served-model-name` alias stable so clients do not change when the weights do. Model swaps often need engine flag changes too: Qwen's model cards, for instance, specify a reasoning parser for vLLM and SGLang.

Before moving traffic, run your evaluation set against the candidate at its deployment precision and compare it with the current model on the same cases; the [LLM evaluation guide](https://theaiengineering.com/articles/llm-evaluation) covers building that set. Then shift traffic gradually. KServe supports canary rollouts for LLMInferenceService by giving two versions the same route group and proportional weights, so a 9-to-1 split sends 10% of traffic to the new model and rolls back by changing one weight.

## Where to start

Start small and let measurements drive each step up. Prototype on a laptop with Ollama, LM Studio, or llama.cpp, confirm with your evals that an open-weights model handles the task, size memory with the formula above, then move to vLLM or SGLang on one GPU server before adding Kubernetes.

A typical rollout order:

1. Pick two or three licensed candidates from the release tracker and evaluate them on your own task at the precision you plan to deploy.
2. Size memory for your target context length and concurrency, and price the hardware at realistic utilization against API costs.
3. Serve the chosen model with vLLM or SGLang on one GPU server, behind a private network and a TLS proxy.
4. Put an AI gateway in front for per-team keys, rate limits, logs, and a fallback route.
5. Wire engine metrics into dashboards and alerts, and load-test at expected peak.
6. Move to Kubernetes when you need more than one replica, autoscaling, or canary rollouts.

## Frequently asked questions

### How much VRAM do I need to run an LLM locally?

Start with the weights: parameters times bytes per parameter. An 8B model needs about 16 GB at 16-bit precision and about 5 GB at 4-bit; Ollama lists qwen3:8b at 5.2 GB and qwen3:32b at 20 GB in their default 4-bit builds. Then add KV cache for your context length and concurrency, plus headroom for the runtime. A model that barely fits leaves no room for long prompts.

### Is it cheaper to self-host an LLM than to use an API?

It depends on utilization. A GPU costs the same per hour whether it serves one request or thousands, while API prices scale with tokens. Divide the hourly cost of your GPUs and supporting infrastructure by the tokens they actually serve per hour at your latency target, then compare that cost per million tokens with API prices for the same input and output mix. Idle hours raise the self-hosted figure quickly.

### Can you run an LLM without a GPU?

Yes. llama.cpp runs GGUF models on x86, ARM, and RISC-V CPUs, and Ollama documents a CPU-only Docker setup. Apple silicon Macs run models on the integrated GPU through Metal. CPU inference is slower because generating each token means reading the weights from memory, and CPUs have less memory bandwidth than GPUs, so small models and 4-bit quantization keep response times usable.

### What is the easiest way to run an LLM locally?

Ollama and LM Studio are the shortest paths. With Ollama, install the app or the Linux script, then run a command such as ollama run qwen3:8b; it downloads a 4-bit build and serves an OpenAI-compatible API on port 11434. LM Studio offers the same through a desktop app with a model browser and chat window, plus the lms command-line tool and a headless mode for servers.

### Is a self-hosted LLM private?

A self-hosted LLM keeps prompts and outputs on your infrastructure, but check what each component sends out. vLLM collects anonymous usage data about hardware and configuration by default and stops when VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1 is set. Model downloads reach Hugging Face or a registry unless you pre-stage weights and set HF_HUB_OFFLINE=1. A gateway fallback to a hosted API also sends data outside.

### What is the difference between a local LLM and a self-hosted LLM?

A local LLM runs on the machine of the person using it, such as a laptop running Ollama or LM Studio. A self-hosted LLM is the broader category: any model your team runs on infrastructure it controls, including shared GPU servers and Kubernetes clusters that serve many users through an API. Every local LLM is self-hosted, but most self-hosted LLMs in production are not local.

## Terms in this guide

- [Open-weights model](https://theaiengineering.com/glossary/open-weights-model): An open-weights model is an AI model whose trained parameters are published for download, so anyone can run, fine-tune, or quantize it on their own hardware under the terms of its license.
- [Inference engine](https://theaiengineering.com/glossary/inference-engine): An inference engine is the software that serves a trained language model: it loads the weights, batches incoming requests, manages KV cache memory, runs the forward passes, and streams generated tokens over an API.
- [OpenAI-compatible API](https://theaiengineering.com/glossary/openai-compatible-api): An OpenAI-compatible API is an HTTP API that accepts OpenAI's request and response formats, so code written for OpenAI's SDKs can call another server by changing the base URL, API key, and model name.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Quantization](https://theaiengineering.com/glossary/quantization): Quantization is a model compression technique that represents a neural network's weights, and sometimes its activations or KV cache, with fewer bits, such as 8-bit or 4-bit integers instead of 16-bit floats.
- [GGUF](https://theaiengineering.com/glossary/gguf): GGUF is a binary file format for running LLMs with GGML-based engines such as llama.cpp, packing the weights, often quantized, and all the metadata needed to load the model into one file.
- [PagedAttention](https://theaiengineering.com/glossary/paged-attention): PagedAttention is an attention algorithm, introduced with vLLM, that stores each sequence's KV cache in fixed-size blocks placed anywhere in GPU memory and tracked by a block table, an idea taken from operating-system paging.
- [Tensor parallelism](https://theaiengineering.com/glossary/tensor-parallelism): Tensor parallelism is a model parallelism technique that splits the weight matrices inside each transformer layer across several GPUs, which compute every token together and combine partial results through all-reduce operations.
- [Continuous batching](https://theaiengineering.com/glossary/continuous-batching): Continuous batching is an LLM serving technique that rebuilds the batch at every decode iteration, so finished requests leave at once and queued requests take their place at the next step.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Mixture of experts](https://theaiengineering.com/glossary/mixture-of-experts): Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token.
- [LoRA](https://theaiengineering.com/glossary/lora): LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a pretrained model's weights and trains small low-rank matrices added to selected layers, cutting trainable parameters and memory use.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [Time to first token](https://theaiengineering.com/glossary/time-to-first-token): Time to first token (TTFT) is the elapsed time from submitting a request to an LLM until the first token of the response arrives, covering request queuing, network transit, and prompt prefill.

## Related guides

- [vLLM vs Ollama vs llama.cpp vs SGLang: which engine to use](https://theaiengineering.com/articles/vllm-vs-ollama): vLLM vs Ollama compared with llama.cpp, SGLang, LM Studio, and TensorRT-LLM: scope, hardware, APIs, concurrency, and which engine fits local or production use.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM observability: what to capture, which metrics, and tools](https://theaiengineering.com/articles/llm-observability): LLM observability guide: what to capture in traces and spans, metrics to track, OpenTelemetry GenAI conventions, tool categories, and keeping prompts private.

Part of the [LLM inference and serving](https://theaiengineering.com/topics/llm-inference) topic hub.

## Cite this guide

The AI Engineering Editors. "How to self-host an LLM: hardware, serving engines, and a production setup." The AI Engineering, updated Nov 18, 2026. https://theaiengineering.com/articles/self-hosted-llm

- Title: How to self-host an LLM: hardware, serving engines, and a production setup
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-11-18
- Updated: 2026-11-18
- URL: https://theaiengineering.com/articles/self-hosted-llm
- Markdown: https://theaiengineering.com/articles/self-hosted-llm.md

To cite one section, link to its anchor:

- [What is a self-hosted LLM?](https://theaiengineering.com/articles/self-hosted-llm#what-is-a-self-hosted-llm)
- [Why self-host an LLM, and when not to](https://theaiengineering.com/articles/self-hosted-llm#why-self-host-an-llm-and-when-not-to)
- [How to choose a model to self-host](https://theaiengineering.com/articles/self-hosted-llm#how-to-choose-a-model-to-self-host)
- [How much GPU memory does a self-hosted LLM need?](https://theaiengineering.com/articles/self-hosted-llm#how-much-gpu-memory-does-a-self-hosted-llm-need)
- [How to run an LLM locally on a laptop or workstation](https://theaiengineering.com/articles/self-hosted-llm#how-to-run-an-llm-locally-on-a-laptop-or-workstation)
- [How to serve a self-hosted LLM from a single GPU server](https://theaiengineering.com/articles/self-hosted-llm#how-to-serve-a-self-hosted-llm-from-a-single-gpu-server)
- [How to self-host LLMs on Kubernetes](https://theaiengineering.com/articles/self-hosted-llm#how-to-self-host-llms-on-kubernetes)
- [Production concerns for a self-hosted LLM](https://theaiengineering.com/articles/self-hosted-llm#production-concerns-for-a-self-hosted-llm)
- [Where to start](https://theaiengineering.com/articles/self-hosted-llm#where-to-start)
- [Frequently asked questions](https://theaiengineering.com/articles/self-hosted-llm#faq)
