---
title: "What is LLM load balancing? Keys, regions, providers"
description: "LLM load balancing spreads model requests across API keys, deployments, regions, or providers by weight, latency, or quota. How it works and vs LLM routing."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/llm-load-balancing"
markdown_url: "https://theaiengineering.com/glossary/llm-load-balancing.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-11"
keywords: ["LLM load balancing","LLM load balancer","model load balancing"]
citation: "The AI Engineering Editors. \"What is LLM load balancing? Keys, regions, providers.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/llm-load-balancing"
---

# What is LLM load balancing? Keys, regions, providers

Also called LLM load balancer, model load balancing.

> **Definition of LLM load balancing:** LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.

LLM load balancing is the practice of spreading requests for a model across several deployments that can each serve it: multiple API keys or accounts, regions of one cloud provider, different providers hosting the same open-weights model, or replicas of a self-hosted server. It raises total throughput above any one deployment's [rate limits](https://theaiengineering.com/glossary/rate-limiting) and keeps traffic flowing when one deployment slows down. Load balancing usually runs in an [AI gateway](https://theaiengineering.com/glossary/ai-gateway) or in the serving layer in front of self-hosted models.

## How LLM load balancing works

LLM load balancing works by keeping a pool of deployments for each model name and choosing one for every request according to a strategy. Common strategies use static weights, observed latency, in-flight requests, or remaining token and request quota. A deployment that keeps failing leaves the pool for a cooldown period and rejoins when the period ends.

Gateway documentation shows the range of strategies:

- [Bifrost](https://docs.getbifrost.ai/features/keys-management) picks among a provider's API keys by weighted random selection, so a key with weight 0.7 receives about 70% of traffic, and uses a key with weight 0 only as a last resort. Each key can carry its own model allowlist, and a request moves to an untried key after a rate limit or a rejected credential.
- [LiteLLM's router](https://docs.litellm.ai/docs/routing) defaults to `simple-shuffle`, which picks a deployment by weight or by its configured requests and tokens per minute. It also offers least-busy, latency-based, usage-based, and cost-based strategies.
- [Portkey](https://portkey.ai/docs/product/ai-gateway/load-balancing) normalizes target weights into traffic shares and balances across API keys, providers, or models, with optional sticky routing on a field such as a user ID.
- [Azure API Management](https://learn.microsoft.com/en-us/azure/api-management/backends) groups backends into pools with round-robin, weighted, and priority-based options, and sends traffic to a lower-priority group only when circuit breakers have tripped on every backend in the higher ones.

For self-hosted models, the Kubernetes [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) picks a replica using data from the model servers, such as prefix cache status and LoRA adapter availability.

| Strategy | Picks the deployment with | Suits |
|---|---|---|
| Weighted or round-robin | A fixed share of traffic | Equal capacity, gradual migrations |
| Least busy | The fewest in-flight requests | Uneven request sizes |
| Latency-based | The lowest recent latency | Mixed regions or providers |
| Usage-based | The most remaining quota | Keys with tight per-minute limits |
| Priority | The highest tier still available | Primary capacity with overflow backups |

## LLM load balancing vs LLM routing

LLM load balancing chooses which deployment serves a model the caller has already selected, while [LLM routing](https://theaiengineering.com/glossary/llm-routing) chooses which model should answer. Routing is a quality and cost decision. Load balancing is an availability and throughput decision, so a pool should contain only deployments that return equivalent output for the same request.

When several providers host the same open-weights model, confirm that they serve the same model version and settings before pooling them. Load balancing also differs from [model failover](https://theaiengineering.com/glossary/model-failover), which moves a request after it fails. Gateways such as Bifrost and LiteLLM configure both, so failover catches the errors that balancing cannot prevent.

Even spreading has a cost for caching. A prompt prefix cached at one provider or replica is not available at another, so [prompt caching](https://theaiengineering.com/glossary/prompt-caching) hit rates fall as traffic spreads. Sticky routing by user or session keeps related requests on one deployment, and cache-aware replica selection sends a request to the server that already holds its prefix in the [KV cache](https://theaiengineering.com/glossary/kv-cache). The [LLM routing guide](https://theaiengineering.com/articles/llm-routing) covers load balancing together with fallbacks and model selection.

## Related terms

- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.

## Guides that cover llm load balancing

- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LLM load balancing? Keys, regions, providers." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/llm-load-balancing

- Title: What is LLM load balancing? Keys, regions, providers
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-11
- URL: https://theaiengineering.com/glossary/llm-load-balancing
- Markdown: https://theaiengineering.com/glossary/llm-load-balancing.md
