Kill switches: granularity is the design

Lesson 4 of 5 in Rollout and Kill Switches: Controlled Exposure.

A kill switch is a control that stops an agent from acting, immediately, without a deploy. That last clause is the whole engineering requirement. If stopping your agent means reverting a commit, waiting for a build, and pushing to production, then your kill switch has a mean time to activation of twenty minutes and you do not have one.

The mistake almost everyone makes is building exactly one — a global off — and calling it done. Then the first real incident arrives as one tenant’s runs are corrupting their CRM records and your only option is to stop the product for all 4,000 customers. So you hesitate. Hesitation is the failure the granularity is there to prevent: the switch you can afford to pull is the switch that actually gets pulled.

Kill-switch granularity, widest blast radius first

Global — all agents, all tenants, off
The big red button: every agent stops accepting work. Reach for it when you do not yet know the scope — a model provider returning corrupted output, a suspected compromise of the orchestrator, a data-integrity incident with unknown reach. Cost: total loss of product function, so it is the switch teams talk themselves out of using. Keep it, own it, and make sure using it is permitted — the on-call engineer must have standing authority to pull it without hunting for a VP at 03:00.
Per-tenant — one customer, workspace, or region
Stop the agent for one tenant while everyone else keeps running. This is the switch that makes incident response proportionate, and in multi-tenant products it is the one you will use most. It also has a second life as a customer-facing control: enterprise buyers increasingly want their own off switch, and giving them one is cheaper than being the reason they cannot get one. Requires tenant identity to be a first-class dimension in your runtime, not a field buried in a request payload.
Per-agent (or per-version) — this agent, or this version of it, off
Disable one agent — refund-handler off, ticket-triage keeps running — or one version, which is just your rollback lever named honestly. Version-level granularity is what lets you halt a canary in seconds while leaving the incumbent serving. Requires agent and version identity on every run and a routing layer that can be re-pointed at runtime.
Per-tool — the capability, not the agent
The most useful and most under-built tier: revoke one tool across all agents and let everything else continue in a degraded but functional state. Refunds off, reads on. Email send off, drafting on. This is least privilege expressed as a runtime dial — the same registry that scopes tools per agent is the natural place to switch one off globally. Effect: the agent keeps working on everything that does not need the revoked capability, which is usually most of its job.
Per-action limit — not off, but bounded
The finest tier, and the one that prevents most incidents from needing a switch at all: caps and gates on individual actions. Refunds above €100 route to a human, no more than 10 recipients per email, at most 50 write operations per hour per tenant, per-run cost ceilings. It degrades capability instead of removing it. Watch: these must be enforced in the runtime, never in the prompt — a limit the model can be argued out of is not a limit.

Build these as runtime-checked configuration, not code paths. The canonical shape is a small state store the runtime consults on every loop iteration and before every effectful tool call: a row per switch, an operator-facing toggle, a change log, and a cache with a time-to-live measured in seconds rather than minutes.

Two design consequences follow, and both are learned the hard way.

Fail closed on the switch check. If the runtime cannot reach the switch store, it must behave as if the answer might be "stop" for effectful actions. A kill switch whose default under failure is "keep going" is optimistic precisely when the system is already unhealthy.

Do not put the switch behind the thing it turns off. If the control plane your operators use to flip switches runs on the same cluster, the same queue, or the same identity provider as the agents, then a bad enough incident takes out the brakes and the engine together. The switch needs an independent path — a separate store, a documented CLI, or a break-glass procedure that works when the dashboard is down.

A kill switch you have never exercised is a hope, not a control

Every untested switch fails the same way in its first real incident: the toggle flips, the dashboard turns green, and the agent keeps going — because a worker cached the config for fifteen minutes, or the queue still held 400 accepted jobs, or the switch stopped new runs but not the 40 mid-flight ones holding open database transactions.

So drill it, on a schedule, in production. Pull one switch, in business hours, with the team watching, and measure three numbers: time from decision to activation, time until the last in-flight run actually stops, and what users saw during the gap. Then write those numbers down as your real recovery capability — because they are, whatever the runbook claims. A drill that surfaces a fifteen-minute config cache has paid for itself several times over.

What "off" must mean for in-flight runs — decide this explicitly

A switch has to specify its behaviour for work already started, and there are only three honest choices.

Drain: accept no new runs, let in-flight ones finish. Fine for cost or quality problems, wrong for anything doing damage — the damage is being done by the in-flight runs.

Freeze: in-flight runs stop before their next effectful call and park in a resumable state. The best default for data-integrity incidents, and the most work to build, because your runtime needs a durable per-run pause point.

Abort: in-flight runs are terminated immediately, mid-trajectory. Fastest and least clean — you must decide what happens to a run that already sent two of the three emails it planned. Whichever you pick, name it in the switch definition; "we turned it off" without a drain semantic is not a defined operation.

Who may pull it, and how fast

Kill-switch authority should be wide and its audit trail tight. Every on-call engineer can pull the global switch; support leads can pull a per-tenant switch; the automated canary monitor can pull a per-version switch with no human at all. Every pull writes who, when, which switch, and the stated reason.

The anti-pattern is authority so narrow that the switch cannot be used at the speed it was built for. If pulling it requires an executive on a phone call, your effective mean time to activation is however long it takes to find that person — measure that number, not the API latency.

Automated halt vs human halt — you need both

Automated halts fire on defined metric breaches within seconds and are the only realistic answer for canaries, where the whole point is to stop before a human notices. Human halts handle everything the metrics do not encode: a customer email that smells wrong, a journalist’s question, a pattern an engineer recognises from last quarter.

Keep them on the same mechanism so they behave identically, log identically, and get drilled identically. And make the automated halt conservative: an over-eager auto-halt trains the team to disable it, which is how you end up with no automation at all.

Re-arming after a pull — the forgotten half

Turning the agent back on deserves as much design as turning it off. Require an explicit re-arm action with a stated reason, never a timeout that silently restores service — an auto-expiring kill switch means an unresolved incident quietly resumes at 04:00.

Re-entry should also drop a rung: a version halted at canary comes back at canary with a fresh soak window, not at full traffic. And if the switch was pulled for a data-integrity reason, the re-arm checklist includes cleaning up what the agent did before it stopped.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.