---
title: "What is model failover? LLM fallbacks explained"
description: "Model failover sends an LLM request to a backup model or provider when the primary fails. How fallback chains work, when to retry instead, and the quality risk."
type: "glossary-term"
canonical_url: "https://theaiengineering.com/glossary/model-failover"
markdown_url: "https://theaiengineering.com/glossary/model-failover.md"
topic: "AI gateways and LLM routing"
author: "The AI Engineering Editors"
publisher: "The AI Engineering"
updated: "2026-11-11"
keywords: ["Model failover","LLM fallback","model fallback","provider failover"]
citation: "The AI Engineering Editors. \"What is model failover? LLM fallbacks explained.\" The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/model-failover"
---

# What is model failover? LLM fallbacks explained

Also called LLM fallback, model fallback, provider failover.

> **Definition of Model failover:** Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.

Model failover is the automatic rerouting of a failed LLM request to a backup target: another deployment of the same model, the same model at another provider, or a different model. Providers have outages, capacity shortfalls, and [rate limits](https://theaiengineering.com/glossary/rate-limiting), so an application with a single target fails whenever that target does. Failover is usually configured as an ordered fallback chain in an [AI gateway](https://theaiengineering.com/glossary/ai-gateway), an agent framework, or the application's model client.

## How model failover works

Model failover works by classifying each failed response, retrying transient errors on the same target, and moving to the next target in the chain when retries run out or the error will not clear. Record which model served each request, because a successful fallback otherwise looks identical to a successful call to the primary.

Provider error references show which status codes are transient. Anthropic's [API errors page](https://platform.claude.com/docs/en/api/errors) lists 429 `rate_limit_error`, 500 `api_error` for internal errors, and 529 `overloaded_error` when the API is temporarily overloaded. OpenAI's [error codes guide](https://developers.openai.com/api/docs/guides/error-codes) uses 429 for both rate limits and quota problems and 503 for a temporarily overloaded model. Both providers separate the two kinds of 429: OpenAI says retrying billing, spend, or quota errors won't restore access, and Anthropic's spend-cap 429 carries no `retry-after` header.

**Figure 1: Retry the primary on a transient error, then fail over to the next target in the chain.**

A gateway retries the primary model after an overloaded error, then fails over to a fallback model and returns its response with the serving model recorded.

Steps:

1. Gateway → Primary model: request
2. Primary model → Gateway: 503 overloaded
3. Gateway → Primary model: retry after backoff
4. Primary model → Gateway: 503 overloaded
5. Gateway → Fallback model: fail over
6. Fallback model → Gateway: 200 response
7. Gateway → Gateway: return it, log model B

Gateways expose the same pattern with different controls:

- [Bifrost](https://docs.getbifrost.ai/features/retries-and-fallbacks) takes a `fallbacks` array of `provider/model` entries, gives each fallback its own retry budget, and rotates to another API key on a 429 before it moves down the chain. The response's routing information names the provider that answered and carries an `is_fallback` flag.
- [LiteLLM](https://docs.litellm.ai/docs/proxy/reliability) retries within a model group up to `num_retries`, then falls back in listed order, with separate chains for context window errors and content policy violations.
- [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/configuration/fallbacks/) tries an ordered list of providers on an error or a request timeout and reports which step answered in a `cf-aig-step` response header.
- [OpenRouter](https://openrouter.ai/docs/guides/routing/model-fallbacks) takes a `models` array, falls back on any error by default, including rate limits, downtime, and moderation flags, and prices the request at the rate of the model that served it.

## Retries vs failover

A retry sends the same request to the same target after a delay, and failover sends it to a different target. Retries with exponential backoff and jitter suit transient failures such as a burst above a rate limit. Failover suits errors that persist on one target, such as an outage, an exhausted quota, or a prompt too long for the context window.

| Error | First response | If it persists |
|---|---|---|
| 429 rate limit | Wait at least `retry-after`, then retry | Fail over |
| 429 quota or spend cap | Do not retry | Fail over or alert |
| 5xx, overloaded, or timeout | Retry with backoff | Fail over after a few attempts |
| Context length exceeded | Do not retry | Fall back to a larger-context model |
| 400 invalid request | Do not retry | Fix the request |

Retries across a long chain multiply load on a struggling provider, so cap total attempts per request and put a failing deployment in cooldown.

## Fallback models and quality risk

A fallback to the same model at another provider mostly changes availability. A fallback to a different model also changes behavior, because prompts tuned for the primary can underperform and the backup may handle [tool calling](https://theaiengineering.com/glossary/tool-calling), [structured outputs](https://theaiengineering.com/glossary/structured-outputs), or long inputs differently. Run every fallback model through the primary's [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation) test set, and alert on fallback rate as well as error rate.

Failover is often confused with two neighboring techniques. [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing) spreads healthy traffic across deployments before anything fails. A [model cascade](https://theaiengineering.com/glossary/model-cascade) also moves a request to another model, but it triggers on a low-quality answer instead of an error. The [LLM routing guide](https://theaiengineering.com/articles/llm-routing) covers fallback chains alongside the other routing strategies.

## Related terms

- [AI gateway](https://theaiengineering.com/glossary/ai-gateway): An AI gateway is a proxy layer between applications and model providers that exposes one API for many models and centralizes routing, failover, access control, cost tracking, caching, and logging.
- [LLM routing](https://theaiengineering.com/glossary/llm-routing): LLM routing is the practice of directing each request to one of several language models or providers, based on rules or a learned classifier, to balance cost, quality, latency, and availability.
- [Rate limiting](https://theaiengineering.com/glossary/rate-limiting): Rate limiting is the enforcement of a maximum number of requests or tokens a client may send to an API within a time window, with excess requests rejected, usually with HTTP 429.
- [LLM load balancing](https://theaiengineering.com/glossary/llm-load-balancing): LLM load balancing is the distribution of requests for one model across several interchangeable deployments, such as API keys, regions, providers, or replicas, by weight, latency, load, or remaining quota.
- [Model cascade](https://theaiengineering.com/glossary/model-cascade): A model cascade is an LLM serving pattern that calls models in order from cheapest to most capable and returns the first answer that passes a quality check, escalating only when it fails.
- [LLM evaluation](https://theaiengineering.com/glossary/llm-evaluation): LLM evaluation is the practice of measuring how well a language model or LLM-based application performs on defined tasks, using test datasets, automated graders, model-based judges, and human review.

## Guides that cover model failover

- [LLM routing: strategies, fallbacks, and learned routers](https://theaiengineering.com/articles/llm-routing): LLM routing guide: rule-based and cost-aware strategies, fallbacks, load balancing, semantic and learned routers like RouteLLM, router evals, and pitfalls.
- [What is an AI gateway? How it works and when you need one](https://theaiengineering.com/articles/what-is-an-ai-gateway): An AI gateway is a proxy between your apps and model providers. Learn how it works, its core features, AI gateway vs API gateway, and open source options.
- [AI agent frameworks compared: LangGraph, CrewAI, ADK, and more](https://theaiengineering.com/articles/ai-agent-frameworks): An AI agent frameworks comparison of LangGraph, LangChain, CrewAI, the OpenAI, Claude, and Google SDKs, Pydantic AI, Mastra, and more, by language and license.
- [AI engineer roadmap: skills, projects, and resources by stage](https://theaiengineering.com/articles/ai-engineer-roadmap): A practical AI engineer roadmap in seven stages, each with the skills to learn, a project to ship, and resources worth the time, from Python to production.
- [What is AI engineering? The discipline, the stack, and the work](https://theaiengineering.com/articles/what-is-ai-engineering): What is AI engineering? A plain definition, how it differs from ML and software engineering, the layers of the stack, and what AI engineers do day to day.
- [How to build an AI agent: a step-by-step guide in Python](https://theaiengineering.com/articles/how-to-build-an-ai-agent): Learn how to build an AI agent from scratch in Python: when you need one, tool schemas, the agent loop, stop conditions, guardrails, evals, and tracing.
- [Bifrost vs LiteLLM: architecture, features, and performance compared](https://theaiengineering.com/articles/bifrost-vs-litellm): Bifrost vs LiteLLM compared: a Go single binary against a Python proxy, with deployment, routing, virtual keys, caching, MCP, benchmarks, and migration.

Part of the [AI gateways and LLM routing](https://theaiengineering.com/topics/ai-gateways) topic hub.

## Cite this definition

The AI Engineering Editors. "What is model failover? LLM fallbacks explained." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/model-failover

- Title: What is model failover? LLM fallbacks explained
- Author: The AI Engineering Editors
- Publisher: The AI Engineering
- Updated: 2026-11-11
- URL: https://theaiengineering.com/glossary/model-failover
- Markdown: https://theaiengineering.com/glossary/model-failover.md
