---
title: "LLM routing: strategies, fallbacks, and learned routers"
description: "LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls."
summary: "LLM routing is the practice of choosing which model and deployment serve each request, based on task, cost, latency, and provider health. This guide covers routing strategies, fallbacks and load balancing, semantic caching, learned routers such as RouteLLM, router evaluation, and common pitfalls."
type: "guide"
canonical_url: "https://theaiengineering.com/articles/llm-routing"
markdown_url: "https://theaiengineering.com/articles/llm-routing.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
published: "2026-10-08"
updated: "2026-11-11"
keywords: ["llm routing","model routing","llm routing strategies","llm load balancing"]
citation: "The AI Engineering Editors. \"LLM routing: strategies, fallbacks, and learned routers.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/llm-routing"
---

# LLM routing: strategies, fallbacks, and learned routers

## TL;DR

- LLM routing makes three separate choices for each request: which model should answer, which deployment should serve it, and what to do when the call fails.
- Rule-based routing and fallback chains deliver most of the reliability and cost benefit for most teams, and they are predictable and easy to audit.
- Learned routers such as RouteLLM predict when a cheaper model is good enough; the RouteLLM authors report cost reductions of over 2x in certain cases without lowering response quality.
- A router should be evaluated on a quality-versus-cost curve against a random-mix baseline, using metrics such as performance gap recovered.
- The most common routing failures are silent: fallbacks to weaker models, prompts that do not transfer between models, retry storms, and semantic cache false hits.

LLM routing is the practice of deciding, for every request, which model answers it and which deployment of that model serves it, based on the task, cost, latency, and provider health. LLM routing exists because no single model is the best choice for every request: a frontier model wastes money on simple classification, and a small model fails on hard reasoning. This guide covers routing strategies from simple rules to learned routers such as RouteLLM, [fallbacks](https://theaiengineering.com/glossary/model-failover) and load balancing, semantic caching, how to evaluate a router, and the pitfalls that make routing fail quietly. It is part of our [AI gateways, routing, and caching](https://theaiengineering.com/topics/ai-gateways) hub.

## What is LLM routing?

[LLM routing](https://theaiengineering.com/glossary/llm-routing) is the selection of a model and deployment for each request according to rules, live metrics, or a trained predictor, with the aim of balancing quality, cost, latency, and availability. Routing logic usually lives in an AI gateway, an application's model client, or the serving layer in front of self-hosted models.

The term covers two problems that are worth keeping apart:

- **Model routing:** choosing which model should answer. This is a quality and cost decision, such as sending easy questions to a [small language model](https://theaiengineering.com/glossary/small-language-model) and hard ones to a frontier model.
- **Deployment routing:** choosing which endpoint serves a model that is available from several providers, regions, or replicas. This is an availability and latency decision, usually called [load balancing](https://theaiengineering.com/glossary/llm-load-balancing).

Our explainer on [what an AI gateway is](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers the proxy layer where most teams implement both.

## How does LLM routing work?

LLM routing works as a short decision pipeline in front of the model call. The router selects a model or an ordered list of candidates, then a healthy deployment for that model, then handles failures with retries and fallbacks. Each stage uses different inputs: request content, live deployment metrics, and error types.

**Figure 1: LLM routing makes three separate choices per request: which model, which deployment, and what to do on failure.**

A request passes through model selection and deployment selection to a provider; on an error, failure handling retries or falls back to the next candidate.

Components:

- **Request**
- **Model selection**: Which model should answer? Rules, classifier, or learned router
- **Deployment selection**: Which endpoint serves it? Load balancing by health and limits
- **Provider**: Selected deployment
- **Failure handling**: What happens on error? Retries, cooldowns, fallback chain

Connections:

- Request → Model selection
- Model selection → Deployment selection
- Deployment selection → Provider
- Provider → Failure handling: error
- Failure handling → Deployment selection: retry or fall back

The pipeline can sit in different places. A gateway applies it to all applications at once. An application-level router can use context that never reaches the gateway, such as the user's plan or the current agent step. A serving-layer router in front of self-hosted [serving engines](https://theaiengineering.com/platforms/serving-engines) can use replica-level signals such as queue depth and cache state.

## LLM routing strategies

LLM routing strategies range from static rules to trained predictors. Rule-based routing selects by request metadata; cost- and latency-aware routing selects by live metrics; fallbacks and load balancing protect availability; semantic routers match intent with embeddings; learned routers and [cascades](https://theaiengineering.com/glossary/model-cascade) predict when a cheaper model is good enough. Most production systems combine two or three of these.

| Strategy | Selects by | Needs | Main risk |
|---|---|---|---|
| Rule-based | Feature, tenant, task type, prompt length | Configuration | Rules go stale as models change |
| Cost- or latency-aware | Live price, latency, rate-limit headroom | Per-deployment metrics | Trades quality away if candidates differ |
| Fallback chain | Error or timeout on the primary | Ordered candidate list | Retry storms and silent quality drops |
| Load balancing | Health, in-flight requests, usage | Several deployments of one model | Lost cache locality |
| Semantic routing | Embedding similarity to example utterances | Examples per route | Out-of-domain requests and threshold tuning |
| Learned router | Predicted quality of the cheaper model | Preference or labeled data | Shift between training data and your traffic |
| Cascade | A quality check on the cheap model's answer | A reliable scorer | Added latency when escalating |

### Rule-based routing

Rule-based routing maps request attributes to models: a support classifier goes to a small model, code generation goes to a code-tuned model, and prompts longer than a candidate's context window go to a model that accepts them. Rules are predictable, cheap to run, and easy to audit, which is why most teams should start here and add learned routing only when evals show headroom.

### Cost- and latency-aware routing

When the same model is available from several providers, the router can select the cheapest or fastest healthy deployment using live measurements. This strategy is safe when candidates serve the same model weights. When candidates are different models, a cost-first rule quietly trades quality for savings, so pair it with quality evals.

## Fallbacks and LLM load balancing

Fallbacks and LLM load balancing keep requests flowing when a provider slows down or fails. Load balancing spreads traffic across several deployments of the same model by health, usage, or latency. Fallbacks move a failed request to the next candidate in an ordered chain. Both need retry limits and cooldowns, or they amplify the outage they were meant to absorb.

### Fallback chains

A fallback chain should separate retryable errors from permanent ones. [Rate limits](https://theaiengineering.com/glossary/rate-limiting), timeouts, and provider 5xx errors justify a retry with backoff on the same target, then a move to the next candidate. A context-length error will fail on every model with the same limit, so it should go straight to a model with a larger window. [Bifrost's retry and fallback documentation](https://docs.getbifrost.ai/features/retries-and-fallbacks) shows the pattern: network errors and 5xx responses retry on the same key with exponential backoff and jitter, rate limits and rejected credentials rotate to another key, and each entry in the request's `fallbacks` list gets its own retry budget. The [LiteLLM routing documentation](https://docs.litellm.ai/docs/routing) describes a similar design with per-deployment cooldowns and separate context window fallbacks.

```python
import time

CHAINS = {  # primary first, then fallbacks
    "classify": ["small-model", "mid-model"],
    "chat": ["mid-model", "frontier-model"],
}
RETRYABLE = {408, 429, 500, 502, 503, 504}

def complete(task, messages, call_model, max_attempts=3):
    """call_model(model, messages) returns (status_code, body)."""
    for model in CHAINS.get(task, CHAINS["chat"]):
        for attempt in range(max_attempts):
            status, body = call_model(model, messages)
            if status == 200:
                return model, body          # log which model served it
            if status not in RETRYABLE:
                break                       # permanent here; try next model
            time.sleep(2 ** attempt)        # exponential backoff
    raise RuntimeError(f"no candidate succeeded for task {task!r}")
```

Return and log the model that served the request. Without that field, a fallback to a weaker model looks like a success in every dashboard. A gateway can record it for you: Bifrost returns the serving provider and model with an `is_fallback` flag in each response's routing information.

### Load balancing across deployments

[Bifrost](https://docs.getbifrost.ai/features/keys-management) spreads traffic across a provider's API keys by weight, keeps keys with a weight of 0 as a last resort, and in its enterprise edition can replace static weights with measured ones through adaptive routing. LiteLLM's router illustrates the other common strategies by name: `simple-shuffle`, which its documentation recommends for production; `least-busy`, which picks the deployment with the fewest in-flight calls; `latency-based-routing`; usage-based routing that respects token and request limits; and `cost-based-routing`. For self-hosted models, routing can also use replica state. The Kubernetes Gateway API Inference Extension adds an endpoint picker that selects a replica using metrics such as prefix cache status and LoRA adapter availability, because a replica that already holds a request's prefix in its KV cache can skip recomputing it. Our guide to [LLM inference optimization](https://theaiengineering.com/articles/llm-inference-optimization) covers the serving side.

## Semantic routers and learned routers like RouteLLM

Semantic routers and learned routers select a model from the content of the request. A semantic router compares the request's embedding with example utterances for each route, while a learned router such as RouteLLM predicts whether a cheaper model's answer will be acceptable. Cascades instead call the cheap model first and escalate when its answer fails a check.

### Semantic routing

A semantic router represents each route by example utterances, embeds the incoming request, and selects the route whose examples are most similar. Aurelio Labs' open source Semantic Router library compares the meaning of a request against routes in vector space, and its 1.x line adds calibrated confidence scores. Semantic routing suits intent dispatch, such as sending billing questions to one prompt and model and technical questions to another. It depends on good embeddings and on a confidence threshold that sends unfamiliar requests to a safe default. The vLLM Semantic Router project applies a related idea at the serving layer, selecting or combining models for each call according to a configured policy.

### Learned routers

[RouteLLM: Learning to Route LLMs with Preference Data](https://arxiv.org/abs/2406.18665) by Ong et al. (2024) trains routers that choose between a stronger and a weaker model for each query. The authors trained four router types, a similarity-weighted ranking router, a matrix factorization model, a BERT classifier, and a causal LLM classifier, on [Chatbot Arena](https://theaiengineering.com/glossary/lmarena) preference data with augmentation. The paper reports cost reductions of over 2x in certain cases without compromising response quality, and routers that kept their performance when the strong and weak models were replaced with different models at test time.

Earlier work set up the same trade-off. Hybrid LLM by Ding et al. (2024) routes queries between a small and a large model by predicted difficulty and reports up to 40% fewer calls to the large model with no drop in response quality.

### Cascades

A cascade orders models from cheapest to most capable and stops at the first answer that passes a check.

**Figure 2: A cascade calls the cheaper model first and escalates only when a check rejects the answer.**

A query goes to a small model first; a check returns its answer on a pass or, on a fail, escalates to a large model whose answer is returned.

Components:

- **Query**
- **Small model**: Called first
- **Check answer**: Quality scorer
- **Return answer**
- **Large model**: Escalation only

Connections:

- Query → Small model
- Small model → Check answer
- Check answer → Return answer: pass
- Check answer → Large model: fail
- Large model → Return answer

[FrugalGPT](https://arxiv.org/abs/2305.05176) by Chen, Zaharia, and Zou (2023) is a cascade method trained to select which combinations of models to use for different queries, and it reports matching GPT-4's performance with up to 98% cost reduction in its experiments. Cascades save money when most queries are easy and the check is reliable. They add latency on every escalated query, because the user waits for two model calls.

## Semantic caching vs prompt caching

Semantic caching and prompt caching both cut cost, but they work at different layers. [Semantic caching](https://theaiengineering.com/glossary/semantic-caching) skips the model call entirely when a new request is similar in meaning to a cached one. [Prompt caching](https://theaiengineering.com/glossary/prompt-caching) still calls the model but reuses computation for a repeated prompt prefix. Routing interacts with both, so plan them together.

| | Semantic caching | Prompt caching |
|---|---|---|
| Where it runs | Gateway or application | Provider or inference server |
| What it reuses | A complete earlier response | Computed state for a repeated prompt prefix |
| Match rule | Embedding similarity above a threshold | Exact prefix match |
| Model call on hit | Skipped | Still made, with a freshly generated response |
| Correctness risk | A similar request that needs a different answer | None to output correctness |
| Best fit | Repetitive FAQs and idempotent lookups | Long system prompts, documents, and tool definitions |

Two routing interactions matter. Spreading traffic with the same long prefix across many providers lowers prompt cache hit rates, so sticky routing by prefix or session can lower cost even when it looks less balanced. A semantic cache placed before the router means cached answers bypass routing, so scope caches per tenant and exclude personalized or time-sensitive requests. [Bifrost's semantic cache](https://docs.getbifrost.ai/features/semantic-caching), for example, scopes entries to the virtual key that wrote them and lets each request set its own cache key, expiry, or a stricter similarity threshold through headers.

## How to evaluate an LLM router

Evaluate an LLM router by plotting quality against cost across the router's thresholds and comparing that curve with a random-mix baseline. Random routing between a strong and a weak model traces a straight line between the two models' scores. A useful router sits above that line, recovering most of the strong model's quality with a fraction of its calls.

The RouteLLM paper defines three metrics worth reusing:

- **Performance gap recovered (PGR):** the router's score minus the weak model's score, divided by the strong model's score minus the weak model's score.
- **Average performance gap recovered (APGR):** PGR averaged across the full range of cost levels, a single number for comparing routers.
- **Call-performance threshold (CPT):** the minimum percentage of calls to the strong model needed to reach a target PGR, such as CPT(50%).

The cheapest way to compute these is offline replay. Run every query in your [eval dataset](https://theaiengineering.com/glossary/golden-dataset) through every candidate model once, grade each output with your existing graders, and store quality and cost per query and model. Any routing policy can then be simulated by lookup without new model calls. [RouterBench](https://arxiv.org/abs/2403.12031) applies this idea at scale with a dataset of over 405,000 inference outcomes.

Grade quality with the same evals you use for releases, as described in our guide to [LLM evaluation](https://theaiengineering.com/articles/llm-evaluation), including an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge) where outputs are open-ended. Before full rollout, run the router in shadow mode, logging its choices without acting on them, then canary it on a small share of traffic while watching quality scores and fallback rates.

## LLM routing pitfalls

The most common LLM routing pitfalls are silent rather than loud. Fallbacks to weaker models look like successful requests, prompts tuned for one model degrade on another, retries multiply load on a struggling provider, and semantic caches return confident wrong answers. Each pitfall has a known mitigation, and most depend on logging which model served each request.

- **Prompts that do not transfer:** a prompt tuned for one model can underperform on a fallback. Keep per-model prompt variants for critical paths and include fallback models in evals.
- **Feature mismatch:** fallback targets must support the request's structured outputs, tool definitions, and context length, or the fallback fails in a new way.
- **Retry storms:** retries across a chain multiply traffic. Cap total attempts per request, add jitter to backoff, and use cooldowns or circuit breakers per deployment.
- **Silent quality drops:** alert on fallback rate and on the share of traffic served by each model, not only on error rate.
- **Training-traffic shift:** a learned router trained on public preference data may not match your queries. Re-run the quality-cost curve on your own dataset before trusting published savings.
- **Hidden cost on hard prompts:** [reasoning models](https://theaiengineering.com/articles/reasoning-models) can generate many more output tokens on difficult inputs, so compare cost per successful task rather than price per token.
- **Router overhead:** an LLM-based router adds a model call to every request; an embedding or classifier router adds far less latency.

Where to go next: our explainer on [AI gateways](https://theaiengineering.com/articles/what-is-an-ai-gateway) covers where routing runs in production, and the [AI gateways directory](https://theaiengineering.com/platforms/ai-gateways) lists gateways that implement these strategies, including Bifrost and LiteLLM. To measure whether a router helps, start with our guide to [LLM evaluation](https://theaiengineering.com/articles/llm-evaluation).

## Frequently asked questions

### What is model routing in AI?

Model routing is the practice of sending each request to the model best suited to it, instead of sending all traffic to one model. A router can select by task type, prompt length, required features, cost, latency, or a learned prediction of which model will answer well. The goal is to keep quality high while spending frontier-model prices only on requests that need them.

### How does an LLM router work?

An LLM router inspects each incoming request and returns the model, or ordered list of models, that should serve it. Simple routers apply rules based on metadata such as feature or tenant. Semantic routers compare the request's embedding with example utterances for each route. Learned routers use a trained classifier to predict whether a cheaper model's answer will be acceptable.

### What is RouteLLM?

RouteLLM is an open source framework and 2024 research paper from LMSYS researchers for training routers that choose between a stronger and a weaker LLM per query. The routers are trained on human preference data collected through Chatbot Arena, plus data augmentation. The framework, released under Apache 2.0, includes four router types and tools for evaluating the cost and quality trade-off.

### What is semantic caching in LLM applications?

Semantic caching stores model responses and returns a stored response when a new request is similar in meaning to an earlier one, measured by embedding similarity above a threshold. A cache hit skips the model call entirely, which saves cost and latency. The risk is a false hit, where two requests look similar but need different answers, so thresholds and cache scope need care.

### What does LLM fallback mean?

An LLM fallback is a backup model or provider that receives a request when the primary target fails with an error, a timeout, or a rate-limit response. Fallbacks are usually configured as an ordered chain. Good fallback setups distinguish retryable errors from permanent ones, limit total retries, and log which model actually served each request so quality changes stay visible.

### What are open source LLM routing tools?

Open source routing tools include Bifrost, an AI gateway with fallback chains, weighted load balancing across API keys, and semantic caching; RouteLLM for learned routing between a strong and a weak model; Semantic Router from Aurelio Labs for embedding-based intent routing; vLLM Semantic Router for policy-driven model selection; and the router built into LiteLLM for load balancing and fallbacks.

## Terms in this guide

- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Prompt caching](https://theaiengineering.com/glossary/prompt-caching): Prompt caching is an LLM API and inference-server feature that stores the computed attention state for a repeated prompt prefix, so later requests sharing that prefix skip reprocessing it.
- [Embeddings](https://theaiengineering.com/glossary/embeddings): Embeddings are fixed-length vectors of numbers that represent data such as text or images so that items with similar meaning sit close together, which lets software compare meaning with arithmetic.
- [KV cache](https://theaiengineering.com/glossary/kv-cache): The KV cache is GPU memory that stores the attention key and value vectors already computed for previous tokens, so a transformer generating text computes them once instead of at every step.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [LMArena](https://theaiengineering.com/glossary/lmarena): LMArena, originally Chatbot Arena and branded Arena since January 2026, is a public leaderboard that ranks language models from crowdsourced human votes between pairs of anonymous model responses.
- [System prompt](https://theaiengineering.com/glossary/system-prompt): A system prompt is the set of developer-written instructions sent to a language model ahead of the conversation, defining its role, rules, output format, and tool use for every turn that follows.
- [Golden dataset](https://theaiengineering.com/glossary/golden-dataset): A golden dataset is a curated, versioned set of representative inputs, each paired with an expected output or grading criteria, that a team treats as ground truth when evaluating an LLM application.
- [Context window](https://theaiengineering.com/glossary/context-window): A context window is the maximum number of tokens a language model can take into account in a single request, including the system prompt, conversation history, tool results, and the generated output.
- [Structured outputs](https://theaiengineering.com/glossary/structured-outputs): Structured outputs is an LLM API feature that constrains generation so the model's response always conforms to a developer-supplied JSON Schema, producing output that code can parse without retries.

## Related guides

- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [LLM evaluation: a practical guide to evals, metrics, and judges](https://theaiengineering.com/articles/llm-evaluation): LLM evaluation for engineers: build eval datasets, pick metrics, calibrate LLM-as-a-judge, evaluate RAG and agents, and gate every release in CI.
- [LLM inference optimization: a practical guide for engineers](https://theaiengineering.com/articles/llm-inference-optimization): LLM inference optimization guide: prefill vs decode, TTFT, KV cache and PagedAttention, continuous batching, quantization, speculative decoding, and GPUs.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this guide

The AI Engineering Editors. "LLM routing: strategies, fallbacks, and learned routers." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/articles/llm-routing

- Title: LLM routing: strategies, fallbacks, and learned routers
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Published: 2026-10-08
- Updated: 2026-11-11
- URL: https://theaiengineering.com/articles/llm-routing
- Markdown: https://theaiengineering.com/articles/llm-routing.md

To cite one section, link to its anchor:

- [What is LLM routing?](https://theaiengineering.com/articles/llm-routing#what-is-llm-routing)
- [How does LLM routing work?](https://theaiengineering.com/articles/llm-routing#how-does-llm-routing-work)
- [LLM routing strategies](https://theaiengineering.com/articles/llm-routing#llm-routing-strategies)
- [Fallbacks and LLM load balancing](https://theaiengineering.com/articles/llm-routing#fallbacks-and-llm-load-balancing)
- [Semantic routers and learned routers like RouteLLM](https://theaiengineering.com/articles/llm-routing#semantic-routers-and-learned-routers-like-routellm)
- [Semantic caching vs prompt caching](https://theaiengineering.com/articles/llm-routing#semantic-caching-vs-prompt-caching)
- [How to evaluate an LLM router](https://theaiengineering.com/articles/llm-routing#how-to-evaluate-an-llm-router)
- [LLM routing pitfalls](https://theaiengineering.com/articles/llm-routing#llm-routing-pitfalls)
- [Frequently asked questions](https://theaiengineering.com/articles/llm-routing#faq)
