Degraded modes, drills, and the promotion contract
Lesson 5 of 5 in Rollout and Kill Switches: Controlled Exposure.
Here is the question that decides whether your kill switch is usable: what does a user see in the second after you pull it?
If the answer is "a spinner, then a 500," you have built a switch nobody will pull during business hours. Teams hesitate to stop an agent for the same reason they hesitate to stop a payment system — not because they doubt the incident, but because the alternative to the agent is nothing. Fix that and pulling the switch stops being a heroic decision and becomes an operational one.
A degraded mode is the answer to "the agent is off, and the user still needs the thing." Every agent needs at least one, chosen deliberately and tested, because an untested fallback is another hope.
Deterministic fallback
What the user gets: the old, boring, non-agentic path — the rules engine, the keyword search, the template reply, the static form.
Best for: high-volume, low-complexity work where the agent was buying quality and speed rather than possibility. Support triage falls back to keyword routing; document search falls back to lexical search.
The trap: the deterministic path rots. Nobody exercises it for eight months, a dependency changes underneath it, and it fails on the day you need it. Route a small percentage of live traffic through the fallback permanently — a tax that keeps it alive — and include it in your drills.
Human queue
What the user gets: an honest handoff. "A specialist is picking this up," with the agent’s partial work attached so the human is not starting cold.
Best for: low-volume, high-value work, and anything where a wrong answer is worse than a slow one — refunds, medical or legal adjacent flows, account changes.
The trap: capacity. An agent handling 4,000 tasks a day cannot fail over to a team of six. Compute the fallback load before the incident and know your queue’s real ceiling; a human queue that instantly overflows is a worse user experience than an honest outage, and it silently converts an agent incident into a support incident.
Read-only / suggest-only mode
What the user gets: the agent still reasons, retrieves, and drafts — it just cannot act. Every effectful tool is off; every proposed action becomes a suggestion a human executes.
Best for: the majority of real incidents, because most agent incidents are about actions, not analysis. It preserves most of the value while removing the blast radius, which is why it is the first mode to reach for.
The trap: it must be a real mode in the runtime with the tools genuinely revoked, not a prompt that says "propose only." And someone has to consume the suggestions, or you have built a queue nobody reads.
Honest unavailability
What the user gets: a clear message that the feature is temporarily off, what to do instead, and when to check back.
Best for: genuinely optional capabilities, and any incident where you do not yet trust any output from the system — a suspected compromise, a data-integrity question with unknown scope.
The trap: silence. An unexplained failure generates more support load and less trust than a plain "this is off while we check something." Write the copy in advance, in the runbook, so nobody is drafting customer-facing language at 03:00.
Now put the ladder, the metrics, the switches, and the fallbacks together into one artifact: a promotion contract written before the rollout starts, not argued about during it.
For each rung: what evidence promotes, what breach halts, who decides, how long the soak runs, and what users see if it stops. Written in advance it is an engineering decision. Written during a mixed-signal canary at 17:00 on a Friday it is a negotiation, and the side arguing for promotion always has a deadline.
| Rung | Promote when | Halt on | Who decides | Degraded mode if halted |
|---|---|---|---|---|
CI → Shadow | Regression suite green; cost-per-task within budget on golden tasks; injection probes pass. | Any golden-task regression, or a suite the change made flaky. | Automated — the CI gate. | N/A — no user exposure yet. |
Shadow → Canary | ≥500 mirrored runs; trajectory-feature diffs explained; proposed-effect ledger reviewed by a human for every effectful tool. | An unexplained shift in the terminal-state mix, or any argument-level difference on a destructive action. | Owning engineer plus one reviewer. | N/A — output discarded. |
Canary → Flag | ≥300 runs per slice over a full daily cycle; all four halt metrics inside thresholds. | Success proxy −3pp; escalation ±25% relative; tokens/task +25% mean or +50% p95; any rise in destructive-tool blocks. | Automated halt; promotion needs a human. | Route the slice back to the incumbent version — invisible to users. |
Flag → Full | Every named cohort clean for its own window; support and product have each flipped the off switch once in a drill. | Any cohort breach, or a support-reported pattern the metrics do not explain. | Service owner, with support sign-off. | Cohort flag off → read-only mode, with the human queue for effectful requests. |
Full (steady state) | Nothing to promote — this rung is monitored, not exited. | Standing kill-switch criteria; drift alerts; the incumbent version stays deployable for at least one release cycle. | On-call, with standing authority to pull any switch. | Per-tool switches first, then per-tenant, then global — each with its named fallback. |
Interactive sorting exercise: Eight failures reached real users. Sort each one into the rung that was *designed* to catch it — the cheapest control that would have caught this specific failure before exposure. (Some belong to kill switches: they were never catchable in advance, only stoppable fast.)
The rollback drill — quarterly, in production, in business hours
Pick a real agent. Announce the window to support. Pull one switch. Measure: decision→activation, activation→last in-flight run stopped, and what a user saw during the gap. Then re-arm through the documented procedure.
Three findings are near-universal on a first drill: config caching delays propagation by minutes, nobody was sure who was allowed to pull it, and the degraded mode had rotted. All three are cheap to fix on a Tuesday and expensive to discover during an incident.
The rollback rehearsal — can you actually get the old version back?
"Roll back" assumes the previous version is still deployable. Check that assumption explicitly: is the prior prompt, tool schema set, and model pin still pinned and runnable? Did a migration change the state shape so the old version cannot read the new records? Is the old model version still offered by the provider, or was it deprecated last month?
Rehearse a full rollback at least as often as you rehearse the kill switch, and keep the previous version deployable for at least one release cycle after promotion.
Game-day the mixed signal, not just the outage
Total failures are the easy drill. Practise the ambiguous case instead: hand the on-call engineer a canary report showing +40% cost and flat success, and see whether they can find the decision, the owner, and the thresholds without asking three people.
If the promotion contract is written down, this exercise takes four minutes. If it is not, the exercise is the finding.
What to write down after every halt
Every halt — automated or human — should produce four artifacts: the traces from the halted slice, at least one new offline eval case built from them, a note on whether the threshold that fired was well-calibrated, and an update to the contract if it was not.
This is what makes the ladder a ratchet rather than a treadmill: each halt permanently raises the floor that the next candidate has to clear in CI, one rung below where the problem was found.
Tool: Agent Incident Tabletop — Run the ambiguous canary and the multi-tenant kill-switch call under time pressure in the Incident Tabletop simulator — the version where the metrics disagree and the clock is running.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.