Alerting on distributions, not single runs

What each severity tier is actually for

1. Page a human now — irreversible harm is accumulating

   Reserve the pager for signals where every further minute costs something you cannot undo: an agent writing to production, spend burning at 20× baseline, a guardrail-trip rate spike that suggests an active prompt injection campaign, or a kill switch that needs a human hand. The bar is not “important” — it is irreversible and accumulating.

2. Open a ticket automatically — real regression, business hours

   A genuine, bounded regression: success rate down five points on one cohort, p99 up 40%, one tool’s error rate tripled. It needs an owner and a due date, not a woken engineer. Auto-file it with the cohort filter and three exemplar trace links attached, because the first ten minutes of any investigation is finding those.

3. Land in a digest a human reads — drift, slow trends, sampled judgments

   Weekly judge scores on sampled traffic, intervention and approval-rate trends, new failure clusters, cost per successful task. Nothing here is urgent, all of it is how you notice slow decay. A digest only counts as oversight if someone is scheduled to read it.

4. Live on a dashboard only — context for an investigation in progress

   Charts you look at because something else fired: token mix by model, retrieval hit rates, per-tenant volume. Useful, not alertable. If you cannot name the action a chart would trigger, it belongs here — or nowhere.

Should this signal page someone?

Interactive decision tree — outcomes:

• Page it

  Population signal, accumulating irreversible harm, a documented ten-minute action, and a backtested low false-positive rate. That is the entire qualifying set — and it should stay small enough to list on one page of a runbook.

• Auto-ticket, do not page

  A real regression that keeps until morning. Attach the cohort filter and exemplar traces to the ticket so the investigation starts from evidence. Paging on quality regressions is the fastest route to a team that ignores the pager.

• Do not alert on one run

  Single-run alerts on a non-deterministic system are a noise generator: retries succeed, one timeout means nothing, and the pager teaches people to dismiss it. Convert it to a rate over a window — or, if this run class genuinely must never happen (a write outside the allowlist, a blocked egress attempt), the correct control is a runtime block plus a security event, not a performance alert.

• Backtest before you arm it

  Replay the rule over the last month of history and count the firings. An untested threshold is how alert fatigue starts, and alert fatigue is a safety failure: it is the mechanism by which the one real page gets swiped away. Add a minimum-sample floor and a trailing baseline, then re-run.