Canary metrics and auto-halt
Lesson 3 of 5 in Rollout and Kill Switches: Controlled Exposure.
A canary without automatic halt is a rehearsal for an incident. The whole value of a small slice is that a breach can trigger rollback before a human reads a dashboard — so the thresholds have to be written down, in advance, as numbers.
The classic web canary watches error rate and p99 latency. Both are still necessary and neither is close to sufficient, because an agent can be a total failure while returning HTTP 200 in 900 ms: it answered confidently and wrongly, burned €4 of tokens doing it, or quietly stopped calling the tool that made the answer true.
Four agent-specific signals carry the load. Watch all four; each one is the only place a particular failure mode shows up first.
| Metric | How to compute it cheaply | The failure only it catches | Starter halt rule |
|---|---|---|---|
1 · Task success proxy | A cheap online signal that correlates with the offline score: user accepted the result, no rework request within the session, no follow-up ticket in 24h, structured output passed schema and business validation. | Quality collapse with no errors. HTTP 200, fast, wrong. Nothing else in the stack notices. | Halt if the proxy drops more than 3 percentage points below the incumbent slice, with enough runs to distinguish that from noise. |
2 · Escalation / handoff rate | Count runs terminating in human handoff, refusal, or turn-limit exhaustion, divided by all runs. | Both directions. A rise means the agent got timid or confused and you just moved load onto a support queue that is not staffed for it. A fall means it started answering things it used to escalate — the more dangerous direction, and the one no naive "fewer escalations is better" dashboard will flag. | Halt on a move of more than ±25% relative in either direction. Yes, both directions. |
3 · Token spend per completed task | Sum input, output, and cached tokens per run from your traces; divide by completed tasks, not by runs. Track the p95 as well as the mean. | The economics regression. A candidate that is 2% better and 60% more expensive passes every quality gate you own and destroys the unit economics of the product. | Halt above +25% mean or +50% p95 versus the incumbent slice. Also halt on a hard per-task ceiling, independent of comparison. |
4 · Guardrail trip rate | Count guardrail and policy blocks per run: injection detector fires, egress denials, blocked tool calls, PII filter hits, schema rejections. | Safety-posture drift. A candidate that is now attempting things the policy layer has to stop is behaving differently even if the policy layer catches all of it. Trips are attempts, and attempts are the leading indicator. | Halt on any increase in blocks of a destructive tool; halt above +50% relative on the aggregate trip rate. |
Two structural rules make the difference between a canary that works and a canary that generates noise.
Set a minimum sample before the halt rules arm. Agent metrics on 20 runs are indistinguishable from a coin flip; auto-halt on the third failure of the morning will train your team to ignore the halt. Pick a floor — a few hundred runs per compared metric is a reasonable starting point for a proxy hovering near 90% — and treat everything below it as unarmed.
Set a soak window that covers a full traffic cycle. Most agent failure modes are correlated with input type, and input types have a daily rhythm: batch jobs at 02:00, humans at 10:00, end-of-month reports on the 30th. A canary promoted after 40 clean minutes has tested one input type.
Now the hard case, and the one every team meets: the canary is not clean, and it is not broken either. Mixed signals are the normal outcome. Walk one.
Your canary shows +40% token spend and flat success. Decide.
Interactive decision tree — outcomes:
- Do not decide yet — extend the canary
Hold the slice, extend the soak window to cover a full daily cycle, and if it is safe to do so, widen slightly to accumulate runs faster. Deciding on an underpowered sample is how teams roll back good versions and promote bad ones. Note this as a process finding too: your minimum-sample floor should have kept the halt rules unarmed rather than presenting you an ambiguous verdict.
- Halt and roll back — this is a loop regression
Unbudgeted spend concentrated in the p95 step count means some runs now flail: repeated tool calls, retries, or self-verification that never terminates. Flat success plus a fat tail is the classic signature of a stopping-condition regression, and the tail is where cost and latency incidents live. Roll back, pull the worst offending traces, and turn three of them into offline eval cases before the next attempt.
- Halt and fix at the source — uniform inflation
A uniform per-run increase points at plumbing, not reasoning: a bigger system prompt, more retrieved chunks, a tool that now returns verbose JSON, or lost prompt caching from reordering the prompt prefix. These are cheap to fix and pointless to ship: you are paying 40% for no measured gain. Roll back, fix the source, and re-enter at shadow — this class of change is fully visible one rung down.
- A real trade-off — escalate the decision, do not auto-promote
The change did what it was designed to do; it just did not move your headline proxy. Now it is an economics decision, not an engineering one: compute cost per task at full traffic and put it next to the value of the metric that moved. Document the trade, get the owner who holds the budget to accept it, and promote to a feature flag rather than straight to full so you can still walk it back cheaply.
- Roll back — you are buying nothing
+40% cost, flat success, no other metric moved. However elegant the change, it is a 40% price increase with no measured benefit, and rolling it back is the boring correct answer. Before you retire it, check the hardest task slice specifically: a change that helps only the top decile of difficulty can show flat aggregate success and still be worth shipping — but you have to demonstrate that on a stratified sample, not assert it in a promotion review.
Key terms: canary deployment, auto-rollback, escalation rate, guardrails, token, trace
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.