The Runbook

The production runbook, in dependency order

1. Failover rehearsed — Prove it, don’t assume it

   A failover that has never run is a hypothesis. Schedule drills that execute the real promotion path — evacuate a region, flip to the standby deployment, walk the degradation ladder — and treat every rung taken in a real incident as free drill data. The standby’s eval suite green today is part of the definition of “ready”.

2. Change gated by evals — Nothing ships on vibes

   Every prompt, model, or provider change passes the regression gate before traffic sees it, and monitoring scores a sample of live traffic for Drift — the mechanism that catches vendor-side model changes you never chose. This layer is the Evaluation domain operationalized: Production Evals and Regression Testing.

3. Guardrails and security configured — Defaults are not your policy

   Each platform ships a safety surface with configurable pieces — filter categories, thresholds, PII handling, Prompt injection defenses — and the configured state, not the default state, is what your policy requires. Private networking for sensitive paths belongs here too. The full defense stack is the Security domain’s capstone: Security in Production.

4. Quota and capacity managed — Headroom is provisioned, not hoped for

   Lesson 1, as a standing discipline: Quota dashboards per region and per deployment, alerts tuned to fire before the ceiling, a reserved baseline sized from measured traffic with spillover wired, and standby-region quota pre-arranged for the failover plan above.

5. Costs modeled and metered — The foundation everything prices against

   The three meters — per-token, provisioned units, GPU-hours — understood and re-run against real traffic as it grows, with Utilization tracked honestly and Egress in the model. Every layer above spends money; this layer is where the spending is deliberate. Tools: the Token Cost Estimator and, if you self-host, GPU Sizing.

Quotas monitored — what does “monitored” actually mean?

A per-region, per-deployment view of consumption against allocation, with alerts at a threshold that leaves time to act (raising quota is a request process with an approval loop, not a knob). Track the 429 rate as a first-class metric, and watch the distribution of traffic across routing paths from lesson 1 — a rising spillover share is a capacity decision announcing itself early.

Costs modeled — and re-modeled

The model you built in Cost Modeling decays as traffic grows and shifts shape. Re-run the Token Cost Estimator on current traffic quarterly; recheck the break-even between pay-per-token and reserved capacity whenever volume changes materially. The formula discipline holds: named variables from your own metering, never someone else’s dollar figures.

Capacity reserved — where it matters, only where it matters

A reserved baseline earns its keep only at healthy Utilization — reserve for the steady-state floor of measured demand, spill peaks to on-demand, and treat standby-region capacity as insurance priced at standby_units × unit_rate × hours. Insurance you have never tested (previous lesson) is the most expensive kind: paid for and worthless.

Evals gating deploys — including the deploys you didn’t make

The gate runs on every prompt and model change; monitoring covers the rest. Managed platforms update and retire models on their own schedules, so pin versions where the platform offers it and let Drift detection catch behavior shifts behind a pinned name. The machinery is in Production Evals and Regression Testing.

Guardrails configured — reviewed against policy, not defaults

All three platforms ship configurable safety surfaces (Bedrock Guardrails are built, versioned, and attached; Azure content filtering is attached per deployment with adjustable severity thresholds; Vertex safety filters expose per-category thresholds). The runbook item is a review: does the configured state match your policy, and does someone own the answer? Depth lives in Security in Production.

Failover tested — the item that validates all the others

A drill is the only honest audit of the stack: it exercises standby quota (capacity item), the standby’s eval baseline (evals item), the standby’s guardrail configuration (security item), and the cost of running degraded (cost item). If the drill has a finding in any layer, the runbook was fiction in that layer. Schedule it, run it, keep the findings.

In production

Where the runbook’s items physically live on each platform — the consoles and mechanisms your on-call rotation should already have open:

AWS

Quotas surface in the Service Quotas console and Bedrock’s service-quotas pages, tracked per model and per inference endpoint. Guardrails are first-class objects — created, versioned with a working draft and test window, and attached at inference by ID and version (or invoked standalone via ApplyGuardrail), which makes “guardrails configured” an auditable statement. Cross-Region routing is verifiable after the fact: CloudTrail logs the processing Region per request. Reserved capacity is Provisioned Throughput per Region — including the standby’s.

Azure

Quota is managed as TPM allocation on the quota pages, redistributable across deployments of the same model — the runbook should name who may rebalance it during an incident. Content-filter configurations are created on the portal’s Guardrails + controls page and attached per deployment, so the review item is per-deployment, not per-account. Deployment types encode the routing/residency stance per deployment; provisioned capacity is PTUs, and batch traffic draws a separate enqueued-token quota that deserves its own alert.

Google Cloud

The Cloud Quotas system distinguishes adjustable quotas (request increases through the console) from fixed system limits (redesign, not requests) — the runbook should record which kind bounds each workload. Safety filters expose per-category blocking thresholds alongside non-configurable filters that always apply. Provisioned Throughput is a term commitment in GSUs with per-request control over spillover, and the global endpoint’s documented limitations (tuning, RAG corpora, batch for some model families) are the checklist for whether it may serve as your availability surface.