The rollout ladder
Lesson 1 of 5 in Rollout and Kill Switches: Controlled Exposure.
Your regression suite is green. The new prompt scores better on the golden dataset. Ship it?
Not to everyone. Offline evals measure the agent against traffic you already collected and problems you already thought of. Production supplies neither. So you buy information in increasing increments: each rung of the rollout ladder exposes a little more reality and gives you a cheap way back down.
The ladder has four rungs, and each one is designed to catch a different class of failure. Skipping a rung is not "moving fast" — it is choosing to learn that rung’s lesson from customers instead of from instruments.
The rollout ladder for an agent version
- CI gate: regression suite + evals green
The offline floor. Golden tasks, injection probes, cost budget per task. Cheap, repeatable, and blind to anything your dataset does not contain.
- SHADOW — runs on real traffic, output discarded
The new version sees production inputs and produces a full trajectory. Nothing reaches the user; nothing commits a side effect. You diff its trajectory against the incumbent’s. Catches: quality and behaviour regressions on real inputs.
- Trajectory diff acceptable?
Not "are outputs identical" — they never are. Are the differences explainable, and are the regressions you found ones you accept?
- CANARY — 1–5% of real traffic, auto-halt armed
A small real slice, chosen to be representative, with automatic rollback on metric breach. Catches: cost and latency surprises, real-user weirdness, tool failures under real load.
- Metrics inside thresholds for the full soak window?
Success proxy, escalation rate, tokens per task, guardrail trip rate — all four, held for long enough to cover a daily traffic cycle.
- FEATURE FLAG — named cohorts, instant off
Targeted exposure you control by identity, not by percentage: internal staff, design partners, one tenant, one region. Catches: cohort-specific breakage; gives product and support a switch they can flip without a deploy.
- FULL — new version is the default
Default for everyone, with the previous version still deployable and every kill switch still wired. "Full" is a traffic state, not the end of monitoring.
- Halt: route back to incumbent, keep the traces
Rolling back is a successful outcome of the ladder, not a failure of it. The traces from the halted slice are the most valuable artifact you own — they become new eval cases.
Read the ladder as a sequence of questions, not a sequence of ceremonies.
Shadow asks: does it behave differently? You get full-fidelity real inputs at zero user risk, which makes shadow the only rung where you can compare two versions on the same input. It is the quality rung.
Canary asks: what does it do that we did not model? Real users hold the conversation differently, paste 40-page PDFs, and hit your tools during their own outage. Canary is also the first rung where you learn what the version costs, because shadow token spend is measured on replayed traffic and real users produce longer, messier sessions.
Feature flag asks: who is it wrong for? Percentage-based canaries average across cohorts and hide a version that is fine for 95% of tenants and catastrophic for the enterprise account with a custom tool. Flags let you name the exposure — and give a non-engineer an instant off switch.
| Rung | Designed to catch | Blind to | Exit criterion |
|---|---|---|---|
Shadow | Quality and trajectory regressions on real inputs: new loops, dropped tool calls, worse final answers, changed refusal behaviour. | Anything that needs a real side effect or a real user reaction. Cost under real session lengths. Load interactions. | Trajectory diff reviewed on a representative sample; every material difference has an explanation you accept. |
Canary | Token spend and latency surprises, real-user inputs no dataset contained, tool and dependency failures under production conditions. | Rare cohorts and low-frequency tasks — 2% of traffic may contain zero instances of your highest-risk workflow. | All halt metrics inside thresholds across a full soak window (at least one complete daily traffic cycle). |
Feature flag | Cohort-specific breakage: one tenant’s custom tools, one region’s language mix, one plan tier’s data volume. | Aggregate effects that only appear at full traffic — rate limits, cache behaviour, queue depth, model provider throttling. | Each named cohort clean for its own window; support and product have exercised the off switch at least once. |
Full | Scale effects, slow drift, and the long tail — the failures that need volume or weeks to appear. | Nothing, and that is the point: at full exposure your only remaining instruments are monitoring and the kill switches. | There is no exit. There is only the next version, entering at the bottom of the ladder. |
Key terms: shadow mode, canary deployment, feature flag, kill switch, blast radius, auto-rollback
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.