What is model failover? LLM fallbacks explained
Also called LLM fallback, model fallback, provider failover
Definition
Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.
Model failover is the automatic rerouting of a failed LLM request to a backup target: another deployment of the same model, the same model at another provider, or a different model. Providers have outages, capacity shortfalls, and rate limits, so an application with a single target fails whenever that target does. Failover is usually configured as an ordered fallback chain in an AI gateway, an agent framework, or the application's model client.
How model failover works#
Model failover works by classifying each failed response, retrying transient errors on the same target, and moving to the next target in the chain when retries run out or the error will not clear. Record which model served each request, because a successful fallback otherwise looks identical to a successful call to the primary.
Provider error references show which status codes are transient. Anthropic's API errors page lists 429 rate_limit_error, 500 api_error for internal errors, and 529 overloaded_error when the API is temporarily overloaded. OpenAI's error codes guide uses 429 for both rate limits and quota problems and 503 for a temporarily overloaded model. Both providers separate the two kinds of 429: OpenAI says retrying billing, spend, or quota errors won't restore access, and Anthropic's spend-cap 429 carries no retry-after header.
Gateways expose the same pattern with different controls:
- Bifrost takes a
fallbacksarray ofprovider/modelentries, gives each fallback its own retry budget, and rotates to another API key on a 429 before it moves down the chain. The response's routing information names the provider that answered and carries anis_fallbackflag. - LiteLLM retries within a model group up to
num_retries, then falls back in listed order, with separate chains for context window errors and content policy violations. - Cloudflare AI Gateway tries an ordered list of providers on an error or a request timeout and reports which step answered in a
cf-aig-stepresponse header. - OpenRouter takes a
modelsarray, falls back on any error by default, including rate limits, downtime, and moderation flags, and prices the request at the rate of the model that served it.
Retries vs failover#
A retry sends the same request to the same target after a delay, and failover sends it to a different target. Retries with exponential backoff and jitter suit transient failures such as a burst above a rate limit. Failover suits errors that persist on one target, such as an outage, an exhausted quota, or a prompt too long for the context window.
| Error | First response | If it persists |
|---|---|---|
| 429 rate limit | Wait at least retry-after, then retry |
Fail over |
| 429 quota or spend cap | Do not retry | Fail over or alert |
| 5xx, overloaded, or timeout | Retry with backoff | Fail over after a few attempts |
| Context length exceeded | Do not retry | Fall back to a larger-context model |
| 400 invalid request | Do not retry | Fix the request |
Retries across a long chain multiply load on a struggling provider, so cap total attempts per request and put a failing deployment in cooldown.
Fallback models and quality risk#
A fallback to the same model at another provider mostly changes availability. A fallback to a different model also changes behavior, because prompts tuned for the primary can underperform and the backup may handle tool calling, structured outputs, or long inputs differently. Run every fallback model through the primary's LLM evaluation test set, and alert on fallback rate as well as error rate.
Failover is often confused with two neighboring techniques. LLM load balancing spreads healthy traffic across deployments before anything fails. A model cascade also moves a request to another model, but it triggers on a low-quality answer instead of an error. The LLM routing guide covers fallback chains alongside the other routing strategies.
Related terms
Guides that cover model failover
- LLM routing: strategies, fallbacks, and learned routers
- What is an AI gateway? How it works and when you need one
- AI agent frameworks compared: LangGraph, CrewAI, ADK, and more
- AI engineer roadmap: skills, projects, and resources by stage
- What is AI engineering? The discipline, the stack, and the work
- How to build an AI agent: a step-by-step guide in Python
- Bifrost vs LiteLLM: architecture, features, and performance compared
Cite this definition
The AI Engineering Editors. "What is model failover? LLM fallbacks explained." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/model-failover
- Title
- What is model failover? LLM fallbacks explained
- Author
- The AI Engineering Editors
- Publisher
- The AI Engineering
- Updated