What is a model cascade? LLM cascades vs routing
Also called LLM cascade, cascading, model cascading
Definition
A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
A model cascade is a sequence of language models, ordered from cheapest to most capable, in which each query goes to the first model and moves to the next only when the answer fails a check. When most queries in a workload are easy, a small language model answers them, and the expensive model, often a reasoning model, runs only on the rest. Cascades are one of the two main ways to cut model cost by query difficulty; the other is LLM routing.
How a model cascade works#
A model cascade works by pairing each model with a scorer and a threshold. The cascade calls the first model, scores its answer, and returns the answer if the score clears the threshold; otherwise it calls the next model. The scorer can be a trained classifier, an agreement check across several sampled answers, or an LLM-as-a-judge.
FrugalGPT (Chen, Zaharia, and Zou, 2023) presents the LLM cascade as one of three cost-reduction strategies, alongside prompt adaptation and LLM approximation. Its scoring function rates how reliable an answer is for a query, and the order of models and the threshold for each stage are learned together under a cost budget; in one case study the scorer is a DistilBERT regression model. The authors report matching GPT-4's performance with up to 98% cost reduction, or 4% higher accuracy at the same cost, in their experiments.
Yue et al. (2023) use the weaker model's answer consistency across samples as a signal of question difficulty: consistent answers stay with the cheaper model, and inconsistent ones escalate. With GPT-3.5-turbo as the weaker model and GPT-4 as the stronger one, they report performance comparable to GPT-4 at 40% of its cost on six reasoning benchmarks.
Model cascade vs LLM routing#
A model cascade decides after seeing an answer, and a router decides before any model runs. Routing picks one model per query from the prompt and its metadata, so each query costs one call. A cascade judges the actual output, which gives it more information, but every escalated query pays for at least two calls and waits for both.
Dekoninck, Baader, and Vechev (2024) define routing as choosing a single model per query and cascading as running increasingly larger models until a satisfactory answer is found. They propose cascade routing, which combines the two in one strategy, and conclude that good quality estimators are critical to model selection of either kind.
| Routing | Cascade | |
|---|---|---|
| Decision point | Before the first call | After each answer |
| Input to the decision | The prompt and metadata | The prompt and the answer |
| Calls per query | One | One or more |
| Latency on hard queries | One model call | Every call up to the model that passes |
| Main dependency | A difficulty predictor | A reliable answer scorer |
Model failover also moves a request to another model, but it triggers on an error; a cascade triggers on a failed quality check. Evaluate a cascade on your own traffic with LLM evaluation: a lenient scorer returns weak answers without any error, and a strict one escalates so often that the cascade costs more than calling the strong model directly. The LLM routing guide compares cascades with rule-based and learned routers.
Related terms
Guides that cover model cascade
Cite this definition
The AI Engineering Editors. "What is a model cascade? LLM cascades vs routing." The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/model-cascade
- Title
- What is a model cascade? LLM cascades vs routing
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated