What is LLM routing? Techniques and strategies

Also called model routing, LLM router, AI model routing

Definition

LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.

LLM routing is the practice of sending each request to the most suitable model or provider instead of sending all traffic to one model. A router can follow fixed rules, such as sending code tasks to one model and short classification tasks to another, or it can use a learned classifier that predicts which model will answer a given prompt well. Routing usually runs in an AI gateway or in the application's model client.

How LLM routing works#

LLM routing works by inspecting each request, or its metadata, before the model call and selecting a target from a pool of models or deployments. The main strategies:

  • Rule-based routing: route by task type, user tier, prompt length, or required features such as tool use or long context.
  • Learned routing: a small classifier predicts whether a cheaper model can handle the prompt. RouteLLM (Ong et al., 2024) trained routers on human preference data to pick between a strong and a weak model, and reported cost reductions of over 2 times in some settings without lowering response quality.
  • Cascades: try a cheaper model first and escalate when a quality check fails. FrugalGPT (Chen et al., 2023) reported matching GPT-4's performance with up to 98% cost reduction on its benchmarks.
  • Availability routing: load-balance across providers or regions, and fail over when one returns errors or rate limits.
  • Latency routing: send each request to the deployment with the lowest observed latency.

Production setups often combine several strategies, for example rule-based model selection with availability-based failover underneath.

Why LLM routing matters#

LLM routing matters because requests vary in difficulty, and the models that handle hard requests well are usually slower and more expensive than models that handle easy requests well. Sending simple requests to a small language model and reserving a reasoning model for hard ones lowers average cost and latency. Availability routing also limits the impact of a single provider's outage or rate limits.

The main risk is silent quality loss: a router that misjudges difficulty sends hard prompts to a weak model, and nothing errors. Measure routed traffic with LLM evaluation on real prompts, and log which model served each request. The LLM routing guide covers implementation patterns, and the AI gateways hub links related topics.

Cite this definition

The AI Engineering Editors. "What is LLM routing? Techniques and strategies." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/llm-routing

Title
What is LLM routing? Techniques and strategies
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated