---
title: "What is LLM routing? Techniques and strategies"
description: "LLM routing sends each request to the model best suited to it by cost, quality, latency, or availability. Learn how LLM routing works and the main strategies."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/llm-routing"
markdown_url: "https://theaiengineering.com/glossary/llm-routing.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["LLM routing","model routing","LLM router","AI model routing"]
citation: "The AI Engineering Editors. \"What is LLM routing? Techniques and strategies.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-routing"
---

# What is LLM routing? Techniques and strategies

Also called model routing, LLM router, AI model routing.

> **Definition of LLM routing:** LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.

LLM routing is the practice of sending each request to the most suitable model or provider instead of sending all traffic to one model. A router can follow fixed rules, such as sending code tasks to one model and short classification tasks to another, or it can use a learned classifier that predicts which model will answer a given prompt well. Routing usually runs in an [AI gateway](https://theaiengineering.com/glossary/ai-gateway) or in the application's model client.

## How LLM routing works

LLM routing works by inspecting each request, or its metadata, before the model call and selecting a target from a pool of models or deployments. The main strategies:

- **Rule-based routing**: route by task type, user tier, prompt length, or required features such as tool use or long context.
- **Learned routing**: a small classifier predicts whether a cheaper model can handle the prompt. [RouteLLM](https://arxiv.org/abs/2406.18665) (Ong et al., 2024) trained routers on human preference data to pick between a strong and a weak model, and reported cost reductions of over 2 times in some settings without lowering response quality.
- **[Cascades](https://theaiengineering.com/glossary/model-cascade)**: try a cheaper model first and escalate when a quality check fails. [FrugalGPT](https://arxiv.org/abs/2305.05176) (Chen et al., 2023) reported matching GPT-4's performance with up to 98% cost reduction on its benchmarks.
- **Availability routing**: load-balance across providers or regions, and fail over when one returns errors or [rate limits](https://theaiengineering.com/glossary/rate-limiting).
- **Latency routing**: send each request to the deployment with the lowest observed latency.

Production setups often combine several strategies, for example rule-based model selection with availability-based [failover](https://theaiengineering.com/glossary/model-failover) underneath.

## Why LLM routing matters

LLM routing matters because requests vary in difficulty, and the models that handle hard requests well are usually slower and more expensive than models that handle easy requests well. Sending simple requests to a [small language model](https://theaiengineering.com/glossary/small-language-model) and reserving a [reasoning model](https://theaiengineering.com/glossary/reasoning-model) for hard ones lowers average cost and latency. Availability routing also limits the impact of a single provider's outage or rate limits.

The main risk is silent quality loss: a router that misjudges difficulty sends hard prompts to a weak model, and nothing errors. Measure routed traffic with [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation) on real prompts, and log which model served each request. The [LLM routing guide](https://theaiengineering.com/articles/llm-routing) covers implementation patterns, and the [AI gateways hub](https://theaiengineering.com/topics/ai-gateways) links related topics.

## Related terms

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [Semantic caching](https://theaiengineering.com/glossary/semantic-caching): Semantic caching is a technique that returns a stored LLM response when a new query is close in meaning to an earlier one, measured by embedding similarity instead of exact text match.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.

## Guides that cover llm routing

- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is LLM routing? Techniques and strategies." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-routing

- Title: What is LLM routing? Techniques and strategies
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/llm-routing
- Markdown: https://theaiengineering.com/glossary/llm-routing.md
