What is LLM load balancing? Keys, regions, providers

Also called LLM load balancer, model load balancing

Definition

LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.

LLM load balancing is the practice of spreading requests for a model across several deployments that can each serve it: multiple API keys or accounts, regions of one cloud provider, different providers hosting the same open-weights model, or replicas of a self-hosted server. It raises total throughput above any one deployment's rate limits and keeps traffic flowing when one deployment slows down. Load balancing usually runs in an AI gateway or in the serving layer in front of self-hosted models.

How LLM load balancing works#

LLM load balancing works by keeping a pool of deployments for each model name and choosing one for every request according to a strategy. Common strategies use static weights, observed latency, in-flight requests, or remaining token and request quota. A deployment that keeps failing leaves the pool for a cooldown period and rejoins when the period ends.

Gateway documentation shows the range of strategies:

  • Bifrost picks among a provider's API keys by weighted random selection, so a key with weight 0.7 receives about 70% of traffic, and uses a key with weight 0 only as a last resort. Each key can carry its own model allowlist, and a request moves to an untried key after a rate limit or a rejected credential.
  • LiteLLM's router defaults to simple-shuffle, which picks a deployment by weight or by its configured requests and tokens per minute. It also offers least-busy, latency-based, usage-based, and cost-based strategies.
  • Portkey normalizes target weights into traffic shares and balances across API keys, providers, or models, with optional sticky routing on a field such as a user ID.
  • Azure API Management groups backends into pools with round-robin, weighted, and priority-based options, and sends traffic to a lower-priority group only when circuit breakers have tripped on every backend in the higher ones.

For self-hosted models, the Kubernetes Gateway API Inference Extension picks a replica using data from the model servers, such as prefix cache status and LoRA adapter availability.

Strategy Picks the deployment with Suits
Weighted or round-robin A fixed share of traffic Equal capacity, gradual migrations
Least busy The fewest in-flight requests Uneven request sizes
Latency-based The lowest recent latency Mixed regions or providers
Usage-based The most remaining quota Keys with tight per-minute limits
Priority The highest tier still available Primary capacity with overflow backups

LLM load balancing vs LLM routing#

LLM load balancing chooses which deployment serves a model the caller has already selected, while LLM routing chooses which model should answer. Routing is a quality and cost decision. Load balancing is an availability and throughput decision, so a pool should contain only deployments that return equivalent output for the same request.

When several providers host the same open-weights model, confirm that they serve the same model version and settings before pooling them. Load balancing also differs from model failover, which moves a request after it fails. Gateways such as Bifrost and LiteLLM configure both, so failover catches the errors that balancing cannot prevent.

Even spreading has a cost for caching. A prompt prefix cached at one provider or replica is not available at another, so prompt caching hit rates fall as traffic spreads. Sticky routing by user or session keeps related requests on one deployment, and cache-aware replica selection sends a request to the server that already holds its prefix in the KV cache. The LLM routing guide covers load balancing together with fallbacks and model selection.

Cite this definition

The AI Engineering Editors. "What is LLM load balancing? Keys, regions, providers." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/llm-load-balancing

Title
What is LLM load balancing? Keys, regions, providers
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated