---
title: "What is a model cascade? LLM cascades vs routing"
description: "A model cascade sends a query to a cheap LLM first and escalates to a stronger model only when a check rejects the answer. How it works and vs routing."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/model-cascade"
markdown_url: "https://theaiengineering.com/glossary/model-cascade.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-06"
keywords: ["Model cascade","LLM cascade","cascading","model cascading"]
citation: "The AI Engineering Editors. \"What is a model cascade? LLM cascades vs routing.\" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/model-cascade"
---

# What is a model cascade? LLM cascades vs routing

Also called LLM cascade, cascading, model cascading.

> **Definition of Model cascade:** A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.

A model cascade is a sequence of language models, ordered from cheapest to most capable, in which each query goes to the first model and moves to the next only when the answer fails a check. When most queries in a workload are easy, a [small language model](https://theaiengineering.com/glossary/small-language-model) answers them, and the expensive model, often a [reasoning model](https://theaiengineering.com/glossary/reasoning-model), runs only on the rest. Cascades are one of the two main ways to cut model cost by query difficulty; the other is [LLM routing](https://theaiengineering.com/glossary/llm-routing).

## How a model cascade works

A model cascade works by pairing each model with a scorer and a threshold. The cascade calls the first model, scores its answer, and returns the answer if the score clears the threshold; otherwise it calls the next model. The scorer can be a trained classifier, an agreement check across several sampled answers, or an [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge).

[FrugalGPT](https://arxiv.org/abs/2305.05176) (Chen, Zaharia, and Zou, 2023) presents the LLM cascade as one of three cost-reduction strategies, alongside prompt adaptation and LLM approximation. Its scoring function rates how reliable an answer is for a query, and the order of models and the threshold for each stage are learned together under a cost budget; in one case study the scorer is a DistilBERT regression model. The authors report matching GPT-4's performance with up to 98% cost reduction, or 4% higher accuracy at the same cost, in their experiments.

[Yue et al. (2023)](https://arxiv.org/abs/2310.03094) use the weaker model's answer consistency across samples as a signal of question difficulty: consistent answers stay with the cheaper model, and inconsistent ones escalate. With GPT-3.5-turbo as the weaker model and GPT-4 as the stronger one, they report performance comparable to GPT-4 at 40% of its cost on six reasoning benchmarks.

## Model cascade vs LLM routing

A model cascade decides after seeing an answer, and a router decides before any model runs. Routing picks one model per query from the prompt and its metadata, so each query costs one call. A cascade judges the actual output, which gives it more information, but every escalated query pays for at least two calls and waits for both.

[Dekoninck, Baader, and Vechev (2024)](https://arxiv.org/abs/2410.10347) define routing as choosing a single model per query and cascading as running increasingly larger models until a satisfactory answer is found. They propose cascade routing, which combines the two in one strategy, and conclude that good quality estimators are critical to model selection of either kind.

| | Routing | Cascade |
|---|---|---|
| Decision point | Before the first call | After each answer |
| Input to the decision | The prompt and metadata | The prompt and the answer |
| Calls per query | One | One or more |
| Latency on hard queries | One model call | Every call up to the model that passes |
| Main dependency | A difficulty predictor | A reliable answer scorer |

[Model failover](https://theaiengineering.com/glossary/model-failover) also moves a request to another model, but it triggers on an error; a cascade triggers on a failed quality check. Evaluate a cascade on your own traffic with [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): a lenient scorer returns weak answers without any error, and a strict one escalates so often that the cascade costs more than calling the strong model directly. The [LLM routing guide](https://theaiengineering.com/articles/llm-routing) compares cascades with rule-based and learned routers.

## Related terms

- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Small language model](https://theaiengineering.com/glossary/small-language-model): A small language model (SLM) is a language model compact enough to run on a consumer device such as a laptop or phone, which in 2025 generally meant fewer than about 10 billion parameters.
- [Reasoning model](https://theaiengineering.com/glossary/reasoning-model): A reasoning model is a large language model trained, usually with reinforcement learning, to generate intermediate reasoning tokens before its final answer, spending more inference compute to solve multi-step problems.
- [LLM-as-a-judge](https://theaiengineering.com/glossary/llm-as-a-judge): LLM-as-a-judge is an evaluation method in which a language model scores or compares the outputs of an LLM application against a rubric, a reference answer, or another response, replacing or supplementing human graders.
- [Model failover](https://theaiengineering.com/glossary/model-failover): Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.

## Guides that cover model cascade

- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [Reasoning models: how they work and when to use them](https://theaiengineering.com/articles/reasoning-models): Reasoning models explained: how test-time compute and reinforcement learning produce chain-of-thought reasoning, when to use them, and their cost and latency.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is a model cascade? LLM cascades vs routing." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/model-cascade

- Title: What is a model cascade? LLM cascades vs routing
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-06
- URL: https://theaiengineering.com/glossary/model-cascade
- Markdown: https://theaiengineering.com/glossary/model-cascade.md
