Even the most reliable LLM providers experience downtime, sudden rate-limit throttling, and unpredictable P99 latency spikes. If your application relies on a single model endpoint with default HTTP settings, provider downtime directly causes application downtime.
Building production-grade AI applications requires a fault-tolerant architecture. Two core resilience patterns form the backbone of LLM reliability: Retries and Fallbacks. While developers often lump them together, they serve fundamentally different failure modes. Misunderstanding when to retry versus when to fall back can burn your API budget, amplify user latency, or create cascading infinite loops.
Here is how to structure retries and fallbacks for maximum LLM uptime, optimal cost control, and minimal latency.
Defining the Resiliency Patterns: Retry vs Fallback
At an architectural level, the distinction between a retry and a fallback comes down to whether you stay within the same failure domain or cross into a new one.
1. Retry Strategy
A retry re-sends the exact same payload to the same model endpoint at the same provider.
- Primary Purpose: Absorb transient, self-healing network or server blips.
- Target Errors: HTTP 503 Service Unavailable, dropped TCP connections, temporary network timeouts.
- Key Mechanism: Exponential backoff combined with randomized delay jitter.
2. Fallback Strategy
A fallback reroutes the request to an alternative model or provider when the primary target fails to deliver.
- Primary Purpose: Maintain uptime during provider outages, severe rate limits, or context window breaches.
- Target Errors: HTTP 429 Rate Limit Exceeded, 500 Internal Provider Outages, ContextWindowExceededError, or policy blocks.
- Key Mechanism: Dynamic provider switching, priority ordering, and payload normalization.
When to Retry: Handling Transient Failures
Retrying is the first line of defense. However, firing rapid-fire HTTP requests at an already struggling provider will exacerbate server load and guarantee failure.
Best Practices for LLM Retries
1. Apply Exponential Backoff with Jitter
Never use fixed interval retries (e.g., retrying every 500ms). Use exponential backoff so each retry waits progressively longer, combined with random "jitter" to avoid the thundering herd problem:
$$t_{\text{wait}} = \min(t_{\text{max}}, t_{\text{base}} \times 2^{\text{attempt}}) + \text{jitter}$$
2. Restrict Retries to Retryable HTTP Status Codes
Only retry errors that are inherently transient:
- Retryable: HTTP 408 (Request Timeout), HTTP 502 (Bad Gateway), HTTP 503 (Service Unavailable), HTTP 504 (Gateway Timeout).
- Non-Retryable: HTTP 400 (Bad Request / Schema Error), HTTP 401 (Unauthorized), HTTP 404 (Model Not Found). Retrying these without payload changes wastes time and money.
3. Set Strict Attempt Limits & Timeouts
Set a hard maximum retry cap (typically 2 to 3 attempts). In interactive applications like chatbots, a long retry loop degrades user experience worse than a graceful error message.
When to Fallback: Navigating Provider Outages and Limits
When a provider is experiencing a full outage or you have exhausted your API rate limit allowance, retrying the same endpoint will repeatedly fail. This is where fallbacks take over.
Designing an Effective Fallback Chain
Primary: OpenAI (gpt-5) ──(Fails / 429)──> Fallback 1: Anthropic (claude-5-sonnet) ──(Timeout)──> Fallback 2: DeepSeek (deepseek-v4)
To build a reliable fallback chain, keep these three rules in mind:
1. Cross Failure Domains
Falling back from gpt-5 to gpt-5-mini on OpenAI keeps you inside the same infrastructure. If OpenAI’s API gateway goes down, both models fail simultaneously. A true fallback chain spans distinct clouds and providers (e.g., OpenAI → Anthropic → Google Gemini).
2. Account for Payload & Schema Discrepancies
Different providers expect different API structures. OpenAI uses response_format: { type: "json_object" }, whereas Claude uses system prompts or tool declarations for structured outputs, and Gemini uses responseSchema. A fallback layer must automatically translate parameters between schemas.
3. Distinguish Error Types for Specific Fallbacks
Not all fallbacks should trigger on general errors. LiteLLM and modern gateways categorise specialized fallbacks:
- Context Window Fallbacks: Reroute to a large-context model (e.g., Gemini 3.5 Pro) if the prompt exceeds the primary model’s limit.
- Content Policy Fallbacks: Reroute if a safety filter incorrectly flags a benign enterprise prompt.
Architectural Comparison Matrix
Implementation Strategies in Code vs Gateway
Option A: SDK-Level Handlers (Client-Side)
You can manage retries and fallbacks directly in your application code using frameworks like ai-retry (TypeScript) or LiteLLM (Python).
Python Example using LiteLLM:
from litellm import Router
# Define multi-provider models
model_list = [
{
"model_name": "gpt-5",
"litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-..."}
},
{
"model_name": "claude-sonnet-5",
"litellm_params": {"model": "anthropic/claude-5-sonnet-20240620", "api_key": "sk-..."}
}
]
# Configure router with retry limits and fallback chain
router = Router(
model_list=model_list,
num_retries=2, # Retry primary model up to 2 times
fallbacks=[{"gpt-5": ["claude-sonnet"]}], # Fallback to Claude if OpenAI fails
cooldown_time=30 # Temporarily bypass failed provider for 30s
)
# Execute query
response = router.completion(
model="gpt-5",
messages=[{"role": "user", "content": "Analyze enterprise API telemetry."}]
)- Pros: Complete visibility in app code.
- Cons: Requires shipping code changes to update fallback rules; secret keys for every provider must be injected into all application microservices; SDK maintenance overhead.
Option B: API Gateway Layer (Infrastructure-Side)
Managing fallbacks across multiple client microservices introduces duplicate logic and security risks. Moving retry and fallback logic to an unified AI Gateway abstracts failovers entirely.
Five Production Pitfalls to Avoid
- Retrying 429 Rate Limits Indefinitely: If you hit an API quota or concurrency ceiling, rapid retries will prolong your throttling window. Instead, trigger a fallback to an alternative provider immediately.
- Ignoring Streaming State: If an error occurs midway through a response stream, a naïve fallback will start streaming from scratch, outputting duplicate text. Your application must reset the client buffer before streaming from a fallback model.
- Mismatched Model Capabilities: Falling back from a reasoning model (e.g.,
5orDeepSeek-v4) to a lightweight model (e.g.,gpt-5-mini) for complex code synthesis can break downstream business logic. Ensure fallbacks match the necessary reasoning tier. - Missing Circuit Breakers: If a primary provider experiences a multi-hour outage, every user request will waste 5-10 seconds timing out before triggering a fallback. Combine fallbacks with circuit breakers to temporarily trip a provider to "unhealthy" and bypass it instantly.
- Loss of Telemetry: When a fallback occurs, log which model ultimately served the request. Otherwise, latency, token costs, and output variations will be impossible to audit.
Zero-Code Failover with AnyAPI Gateway
Writing manual fallback pipelines across distinct SDKs introduces technical debt. AnyAPI streamlines multi-LLM resilience into a unified control plane.
# AnyAPI Gateway Route Configuration
route: /v1/chat/completions
strategy:
retry:
max_attempts: 2
backoff: exponential
status_codes: [500, 502, 503, 504]
fallback_chain:
- model: openai/gpt-5
timeout_ms: 3000
- model: anthropic/claude-5-sonnet
timeout_ms: 3500
- model: google/gemini-3.5-proWhy Developers Choose AnyAPI for Resilience:
- Unified API Format: Use standard OpenAI-compatible requests. AnyAPI handles translation to Claude, Gemini, or DeepSeek on the fly.
- Automatic Failover: When rate limits or outages strike, AnyAPI redirects requests across cloud providers in milliseconds.
- Integrated Circuit Breakers: AnyAPI continuously monitors health telemetry across top providers, routing traffic away from degraded models before your app even notices.
- Unified Billing & Analytics: Single API key management with real-time tracking of retries, fallbacks, latency, and cost attribution.
Frequently Asked Questions
Q1: Should I retry on HTTP 429 Rate Limit errors?
If the response includes a Retry-After header with a short delay (e.g., <2 seconds), a brief retry with jitter is acceptable. However, if rate limits are hit under heavy application traffic, immediately trigger a cross-provider fallback to preserve responsiveness.
Q2: How do fallbacks affect streaming responses (SSE)?
If an error occurs before bytes are flushed to the client, the gateway can transparently switch providers. If an error occurs mid-stream, the gateway must terminate the stream or signal a client reset before initializing the fallback model.
Q3: Will using a fallback model increase my API latency?
If a fallback is triggered after a primary request times out, total P99 latency will increase by the duration of the timeout setting. You can mitigate this by keeping primary timeouts short (e.g., 2.5s - 3.5s) for fast failover.
Q4: How does AnyAPI handle payload differences between providers?
AnyAPI normalizes system prompts, function calling parameters, context windows, and response formats across OpenAI, Anthropic, Google, and open-source models, preventing payload translation errors during failover.
.svg)
%201.png)
