LLM Fallback Strategies: Building Multi-Model Routing That Survives Outages and Rate Limits

Short answer: Build a cost-ordered chain of models fronted by a router that tries the cheapest capable option first, retries transient failures with exponential backoff, and only fails over to a different model or provider once retries are exhausted. Wrap every provider behind an adapter that normalizes prompts and tool schemas, track per-model health so you skip providers that are actively down, and prove the whole thing works by injecting faults in staging.

Failure modes to design for

Your router has to handle four distinct kinds of failure, and they need different responses:

• 429 rate limits. A 429 is a flow-control signal, not a hard error. The OpenAI cookbook is explicit about this: back off and retry rather than treating it as a dead request. Most 429s clear within a second or two.
• Timeouts. The request hung. Set an aggressive client-side timeout (well below your user-facing SLA) so a slow provider doesn't eat your entire latency budget before you fail over.
• Provider outages. 500s, 503s, connection refused. Retrying the same endpoint is pointless here — this is your cue to switch providers.
• Degraded quality. The response comes back 200 but is empty, malformed JSON, or a refusal. Silent failures. You need output validation to catch these, or they slip through your retry logic entirely.

Order the chain by cost, not capability

The best-practice ordering (well put by buildmvpfast) is try cheaper before you try different. Rank your fallback chain by cost so the common, healthy path is also the cheapest path. A fallback chain only costs more when something fails — on a good day every request lands on step one.

A sensible three-tier chain:

1. Cheap, fast model (e.g. a small model) — handles the bulk of traffic.
2. Same tier, different provider — covers a single-vendor outage without a price jump.
3. Premium model — last resort, accepted cost for staying up.

Layer a health check / circuit breaker on top: if a provider has returned errors N times in a row, mark it unhealthy and skip it for a cooldown window instead of paying the timeout on every request.

Retry with backoff before you fail over

Within a single model, retry transient failures before giving up on it. Exponential backoff with jitter is the recommended first line for rate-limit recovery (OpenAI). Jitter matters: without it, every client that got 429'd at the same instant retries at the same instant and re-synchronizes the stampede.

async def call_with_fallback(request, chain):
    for model in chain:                      # cost-ordered
        if not health.is_ok(model):          # circuit breaker
            continue
        for attempt in range(3):             # retry same model first
            try:
                resp = await model.invoke(request, timeout=8)
                if validate(resp):           # catch degraded output
                    health.record_ok(model)
                    return resp
            except RateLimited:              # 429 → flow control
                await sleep(min(2 ** attempt + random.random(), 10))
            except (Timeout, ProviderError): # don't retry outages
                health.record_fail(model)
                break                        # jump to next model
    raise AllModelsFailed()

Note the two exits: 429s stay on the same model and back off; timeouts and 5xx break out immediately to the next model. Retrying a downed provider three times just adds latency.

Cross-provider fallback: normalize behind an adapter

Falling back across providers only works if every provider gets a request it understands. OpenAI, Anthropic, and Gemini differ in message roles, system-prompt handling, and — most painfully — tool/function schema shapes.

Put an adapter in front of each provider that accepts one internal request format and translates:

• Prompts: map your normalized messages to each provider's role conventions (Anthropic's separate system field vs. an inline system message).
• Tool schemas: translate one canonical tool definition into each provider's function-calling format, and normalize the tool-call response back.
• Outputs: return a single response object so downstream code never branches on which model answered.

Keep the internal format as your source of truth. Adding a fourth provider then means writing one adapter, not touching your business logic.

Test fallbacks with fault injection

Fallback code that never runs is fallback code that doesn't work. Test it deliberately:

• In staging, force 429s and timeouts on the primary and confirm traffic shifts to the next tier.
• Simulate a full provider outage (block the host) and verify the circuit breaker trips and recovers after cooldown.
• Feed malformed responses to confirm your output validation catches degraded quality.
• Watch cost and latency during the drill — a fallback that quietly routes everything to the premium model is a bill waiting to happen.

Order by cost, retry with backoff, fail over on real outages, and inject faults so you find out in staging — not at 3am.

Related: more writing.