What is model failover? LLM fallbacks explained

Also called LLM fallback, model fallback, provider failover

Definition

Model failover is the automatic rerouting of an LLM request to a backup model, deployment, or provider when the primary target returns an error, times out, or hits a rate limit.

Model failover is the automatic rerouting of a failed LLM request to a backup target: another deployment of the same model, the same model at another provider, or a different model. Providers have outages, capacity shortfalls, and rate limits, so an application with a single target fails whenever that target does. Failover is usually configured as an ordered fallback chain in an AI gateway, an agent framework, or the application's model client.

How model failover works#

Model failover works by classifying each failed response, retrying transient errors on the same target, and moving to the next target in the chain when retries run out or the error will not clear. Record which model served each request, because a successful fallback otherwise looks identical to a successful call to the primary.

Provider error references show which status codes are transient. Anthropic's API errors page lists 429 rate_limit_error, 500 api_error for internal errors, and 529 overloaded_error when the API is temporarily overloaded. OpenAI's error codes guide uses 429 for both rate limits and quota problems and 503 for a temporarily overloaded model. Both providers separate the two kinds of 429: OpenAI says retrying billing, spend, or quota errors won't restore access, and Anthropic's spend-cap 429 carries no retry-after header.

A gateway retries the primary model after an overloaded error, then fails over to a fallback model and returns its response with the serving model recorded. GatewayRetry and fallback policyPrimary modelProvider AFallback modelProvider B1request2503 overloaded3retry after backoff4503 overloaded5fail over6200 response7return it, log model B
Figure 1: Retry the primary on a transient error, then fail over to the next target in the chain.

Gateways expose the same pattern with different controls:

  • Bifrost takes a fallbacks array of provider/model entries, gives each fallback its own retry budget, and rotates to another API key on a 429 before it moves down the chain. The response's routing information names the provider that answered and carries an is_fallback flag.
  • LiteLLM retries within a model group up to num_retries, then falls back in listed order, with separate chains for context window errors and content policy violations.
  • Cloudflare AI Gateway tries an ordered list of providers on an error or a request timeout and reports which step answered in a cf-aig-step response header.
  • OpenRouter takes a models array, falls back on any error by default, including rate limits, downtime, and moderation flags, and prices the request at the rate of the model that served it.

Retries vs failover#

A retry sends the same request to the same target after a delay, and failover sends it to a different target. Retries with exponential backoff and jitter suit transient failures such as a burst above a rate limit. Failover suits errors that persist on one target, such as an outage, an exhausted quota, or a prompt too long for the context window.

Error First response If it persists
429 rate limit Wait at least retry-after, then retry Fail over
429 quota or spend cap Do not retry Fail over or alert
5xx, overloaded, or timeout Retry with backoff Fail over after a few attempts
Context length exceeded Do not retry Fall back to a larger-context model
400 invalid request Do not retry Fix the request

Retries across a long chain multiply load on a struggling provider, so cap total attempts per request and put a failing deployment in cooldown.

Fallback models and quality risk#

A fallback to the same model at another provider mostly changes availability. A fallback to a different model also changes behavior, because prompts tuned for the primary can underperform and the backup may handle tool calling, structured outputs, or long inputs differently. Run every fallback model through the primary's LLM evaluation test set, and alert on fallback rate as well as error rate.

Failover is often confused with two neighboring techniques. LLM load balancing spreads healthy traffic across deployments before anything fails. A model cascade also moves a request to another model, but it triggers on a low-quality answer instead of an error. The LLM routing guide covers fallback chains alongside the other routing strategies.

Cite this definition

The AI Engineering Editors. "What is model failover? LLM fallbacks explained." The AI Engineering, updated Nov 11, 2026. https://theaiengineering.com/glossary/model-failover

Title
What is model failover? LLM fallbacks explained
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the AI gateways and LLM routing hub · Updated