Failover Patterns
Lesson 2 of 3 in In Production: Multi-Region, Quota, and Failover.
Failure has layers, and so does the answer. The failover ladder runs from cheap and provider-handled to expensive and entirely yours:
Multi-AZ — surviving the loss of a zone or a host — is mostly not your problem on managed platforms: the model APIs are regional services, and redundancy inside the region is the provider’s job. It becomes your problem the moment you self-host: a Serving engine replica set pinned to one zone’s GPU pool is one zonal event away from zero capacity, so the platform mechanisms you met in the Kubernetes module (node pools across zones, topology spread, surplus replicas) are the floor.
Multi-region — surviving a regional outage or capacity drought — comes in two shapes. Active/active keeps traffic flowing to multiple regions continuously; the cross-region mechanisms from lesson 1 are a managed version of it inside one provider. Active/passive keeps a standby that takes no traffic until promoted — cheaper day to day, and the shape with the famous trap: Quota and reserved capacity are per-region resources, so an unprovisioned standby is a diagram, not a plan.
Multi-provider — surviving a provider-wide outage, a pricing rupture, or an exit — is the top rung and the least mechanical. An abstraction layer translates API shapes; that part is commodity. What makes the rung real is behavioral portability: prompts that have variants per provider, and an Eval harness proving the standby actually does your job. The deep dive below is about why that — not the adapter — is the work.
| Failure scenario | Pattern that answers it | Cost of readiness |
|---|---|---|
Zonal outage / host loss | Managed API: provider-handled inside the regional service. Self-hosted: replicas spread across zones with surplus headroom | Managed: included in the service. Self-hosted: |
Regional capacity drought (throttling, not outage) | Cross-region spreading via the documented mechanisms: Bedrock inference profiles, Azure Global / Data Zone deployments, Vertex multi-region or global endpoints | Little standing cost — the platform routes and you pay normal serving rates (Bedrock documents global profiles at ~10% below standard); the real cost is the residency analysis done up front |
Full regional outage | Active/active across regions, or active/passive with a genuinely provisioned warm standby | Active/active: two regions’ capacity each running below peak Utilization. Active/passive: |
Provider-side model change (silent update, quality regression) | Version pinning where the platform offers it, Regression testing on every change you control, and Drift monitoring for the changes you do not | Engineering time, not capacity: an Eval harness wired into CI and a scored sample of live traffic |
Provider outage or exit | Multi-provider abstraction layer plus portable prompts and evals — with a degradation ladder for the gap while traffic shifts | The highest rung: a second integration kept warm, prompt variants maintained per provider, and the full eval suite kept green on both — paid continuously, not once |
Choosing between the two multi-region shapes is a utilization-versus-certainty trade. Active/active pays for capacity in every region continuously but proves every region works continuously too — the failover path is the normal path, exercised by every request. Active/passive concentrates spend but concentrates risk with it: the promotion path runs rarely, decays silently, and gets its first real test during an incident, exactly when the region you are evacuating into is receiving everyone else’s evacuation too. If you run active/passive, the standby needs three things before the incident: quota raised to your measured peak (per-region, remember), whatever reserved capacity your latency floor requires, and a drill calendar that executes the promotion for real.
And when no region can serve the request at full quality, the answer should still not be a raw error. A degradation ladder is a designed sequence of cheaper answers: a smaller model that your evals already cover, a cached response for repeated queries, and — at the bottom — a graceful, honest refusal that is worded, instrumented, and retry-friendly. Each rung keeps more product value than the one below it; the design work is deciding the rungs per feature ahead of time.
The degradation ladder
- Primary route unavailable or exhausted
Regional outage, provider incident, or every quota meter hot — the router has run out of full-quality options.
- Smaller model available for this feature?
A cheaper, smaller model — same provider or same family — can often carry the feature at reduced quality. Only a real option if your prompts and evals already run against it; an untested fallback is a second incident.
- Serve via fallback model
Tag the response as degraded in your telemetry. The share of traffic on this rung is a first-class incident metric.
- Cached answer usable?
For repeated or near-duplicate queries — FAQ-shaped traffic, popular retrieval results — a cached response beats an error. Mark it as possibly stale wherever freshness matters.
- Serve cached response
Semantic or exact-match caches built in calmer times pay out here. Cache hit rate during incidents is a number worth knowing in advance.
- Graceful refusal
An honest “try again shortly”, queued for retry where the product allows — designed and worded ahead of time, never a raw 500. The bottom rung is still a designed rung.
- Degraded response served
The product stayed up. The postmortem still happens — every rung taken is data for next quarter’s capacity plan.
Why multi-provider failover is mostly an eval problem
The visible difference between providers is the API, so teams budget for an adapter and declare the rung built. But the adapter solves the cheapest difference. The expensive ones are behavioral. Different model families respond differently to identical prompts: instruction-following and output-format adherence vary; refusal boundaries sit in different places; JSON and tool-call reliability differ; each model’s tokenizer counts your traffic differently, shifting cost and truncation points; and each platform runs different content filters with different defaults, so inputs that pass one provider’s safety surface get intercepted by another’s. A failover switch that only swaps endpoints therefore swaps your product’s behavior — silently, mid-incident, in front of users.
What makes the rung real is treating behavior as the portable artifact. That means prompts maintained as per-provider variants under version control (tuned once per provider in calm conditions, not rewritten during an outage), and a provider-neutral Eval harness with per-provider baselines that runs against the standby continuously — because the standby’s model updates on its own schedule whether or not you are watching. The failover criterion stops being “the adapter compiles” and becomes “the standby’s eval suite is green as of this morning.” Regression discipline, gates, and drift monitoring are the Evaluation domain’s machinery — Production Evals and Regression Testing is the module this rung stands on.
In production
The failover primitives each platform actually gives you — and the caveat on each that failover plans most often miss:
AWS
Within a Region, Bedrock’s resilience is the service’s concern; across Regions, inference profiles are the documented spreading mechanism — geographic profiles when residency bounds routing, global when it does not, with the processing Region logged in CloudTrail either way. The planning caveat: inference profiles do not currently support Provisioned Throughput, so reserved capacity is a per-Region asset — an active/passive design needs its own Provisioned Throughput (or verified on-demand quota) purchased in the standby Region.
Azure
Deployment types are the availability dial: Global routes to available datacenters dynamically, Data Zone bounds processing to a US/EU/Asia-Pacific zone, and regional types pin a geography — data at rest stays in the designated geography in all cases. For explicit region-to-region failover, remember quota is per subscription, per region, per model: the standby region needs TPM allocated (and PTUs, if you rely on provisioned capacity) before the incident, and the model must actually be available there — new models reach geography-bound types last.
Google Cloud
The global endpoint is documented to improve availability and reduce 429s — effectively a managed active/active inside the provider — but its documented limitations (no tuning, no RAG corpus, batch and Provisioned Throughput restrictions) mean workloads on tuned models or RAG Engine corpora still need explicit regional failover. Plan those with per-region quota in view, and remember tuned models share the base model’s quota in whatever region they fail over to.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.