Declare early, contain first

Lesson 3 of 5 in Monitoring and Incident Response: When the Agent Is the Incident.

Almost all of SRE incident practice transfers directly: declare early and cheaply, name one incident commander, keep a timestamped log, separate the person fixing from the person communicating, and hold a blameless postmortem. Borrow all of it. You do not need a new discipline for agent incidents; you need one amendment to it.

The amendment is this: in a classical outage the system is broken and therefore mostly harmless. In an agent incident the system is working perfectly and therefore dangerous. A crashed service stops doing damage the moment it crashes. An agent that has misunderstood its task keeps executing, at machine speed, for as long as you leave it running — including through your entire diagnosis call.

Agent incident lifecycle

Signal fires
An alarm from lesson two, a guardrail spike, a customer report, or an engineer noticing something odd in a trace. Treat all four as equal grounds to declare.
Declare: one commander, one channel, one log
Declaring costs a Slack channel. Not declaring costs the timeline you will need for the postmortem. Declare early and stand down loudly if it turns out to be nothing.
Can you bound the blast radius now?
Can you name, with evidence, which tenants, tools and versions are affected — in under two minutes? If not, you cannot contain narrowly, and guessing narrow is worse than acting broad.
Contain broadly: kill switch or credential revoke
Stop everything. Collateral damage to healthy runs is a known, bounded cost; an unbounded incident is not.
Contain narrowly: scoped kill, tool disable, force approval
Cut only the affected slice: one tenant, one tool, one agent version — or drop the autonomy tier so every action needs a human.
Freeze the evidence
Extend trace retention, snapshot the memory store and the resolved config, export the prompt and tool-schema versions. This step is skipped in almost every first agent incident, and it is why the postmortem is guesswork.
Diagnose from traces: what ran, what it saw, what it did
The three forensic questions of lesson four. Reconstruct from recorded evidence, not from the agent’s own account of itself.
Failure mode identified?
Name it from the catalogue. "The model was dumb" is not a failure mode and cannot be fixed, tested, or alarmed on.
Widen: diff against a known-good run
Same task, previous version, or a passing run from the same day. In non-deterministic systems the diff between a good and a bad trace is worth more than either trace alone.
Fix, and verify against the new eval case
Write the failing case first, watch it fail, then fix. A fix with no failing case is a hypothesis you shipped.
Restore autonomy in stages
Re-enable for one tenant, then a percentage, watching the alarm that caught it. Full restoration is a decision with an owner, not the default end of an incident.
Postmortem → eval case, guardrail, permission change
The flywheel of lesson five. An incident that produces only a document will happen again.

"Contain" is not one action. It is a ladder, and the rung you choose trades speed and certainty against collateral damage. The top rung stops everything in seconds and takes healthy work down with it; the bottom rung costs nothing and stops nothing. Pick the highest rung whose collateral damage you can defend, and climb back down as you learn more.

The rungs only exist if someone built them before the incident. Every one of these is a deployment-time capability — a flag, a scoped credential, an autonomy setting — which is why the rollout-and-kill-switches module in this domain is a prerequisite for having a usable incident process at all.

The containment ladder

Global kill switch — every run, every tenant, seconds
Fail the agent loop closed for everyone. Use it when you cannot bound the blast radius, when the failure may be an active attack, or when irreversible actions are in play. Requirements: it must be one action, testable, and exercised in drills — a kill switch nobody has pulled in anger is a hypothesis. In-flight runs must terminate, not merely stop receiving new work.
Scoped kill — one tenant, task type, or agent version
Stop only the affected slice. This is the rung you want most of the time, and it is only available if your runs are labelled with tenant, task type and version — the same dimensions the alarms in lesson two carry. Without those labels, "scoped" is a guess and you belong one rung higher.
Revoke permissions / disable the dangerous tool — agent keeps working, hands removed
Leave the loop running read-only. Extremely useful for reliability incidents where the agent is still delivering value: it can triage, draft and report but not write, send or delete. This is least privilege applied under time pressure, and it doubles as your best evidence-gathering mode — the agent keeps producing traces without producing damage.
Force the approval gate — drop the autonomy tier
Every side-effecting action now needs a human click. Cheap, reversible, and it keeps the service alive — but it moves the failure into a queue and a tired reviewer. Only real containment if the queue volume is something humans can genuinely evaluate; otherwise you have built a rubber stamp. Pair it with a review-time metric so you can see when that happens.
Rate limit and budget cap — slows the bleed, does not stop it
Caps concurrency, turns per run, or spend per tenant at the gateway. Correct as a permanent control and as a stopgap while you build the rung above; wrong as your answer to an active incident, because a bounded rate of irreversible actions is still irreversible actions. Use it to buy minutes, not to close the incident.
Monitor only — a decision, not a default
Sometimes the right call: the impact is bounded, reversible and understood, and stopping the agent costs more than letting it run. It is legitimate only with a named owner actively watching, a written trigger for escalating a rung, and a time limit. "We decided to keep watching it" with nobody watching is how a contained incident becomes tomorrow’s bigger one.

Walk the first five minutes below. Notice that the tree asks about evidence and irreversibility before it asks anything about root cause — because in minute one you do not know the cause, and waiting until you do is the mistake the tree exists to prevent.

The first five minutes of an agent incident

Interactive decision tree — outcomes:

Security containment: revoke, then kill
Suspected injection or poisoning is handled as an attack until proven otherwise. Cut egress and credentials first, then terminate runs, then preserve evidence — reversing those two steps leaves an exfiltration window open for exactly as long as your investigation takes. Hand off to the security incident path and treat data access as compromised until the trace says otherwise.
Global kill switch
You cannot bound it, so do not pretend to. Stop everything, accept the collateral damage to healthy runs as a known cost, and buy yourself an uncontaminated hour to investigate. The scoping you wish you had is a labelling problem — fix it in the postmortem so next time you land a rung lower.
Scoped kill for the affected slice
Irreversible actions plus a bounded blast radius is the textbook case for a scoped kill: stop that tenant, task type or version and leave the rest serving. Then freeze evidence before anything expires. This is the outcome a well-instrumented team reaches most often, and it is only reachable because the alarms carried tenant and version dimensions.
Drop the autonomy tier and watch
Reversible actions with a known scope: force the approval gate or strip write permissions and keep the service alive. You keep producing traces — the best diagnostic material you will get — without producing damage. Set an explicit trigger for climbing a rung and a time limit for the arrangement.
Freeze the evidence first
The agent has stopped, so containment is moot — but evidence decays on a timer. Extend trace retention, export the run set, snapshot the memory store and the resolved config, and record the version tuple before anything rotates or a deploy overwrites it. Investigating with expired traces is how incidents end in "we could not determine what happened".
Straight to diagnosis
Nothing to contain and nothing decaying: go to the forensic questions in the next lesson. Still declare the incident and keep the log — a bounded past incident with a clean trace set is the cheapest learning opportunity your team will get all quarter.

Tool: Agent Incident Tabletop — Run the ladder under time pressure: the Incident Tabletop drops you into a live agent incident with partial evidence, a spending graph moving in the wrong direction, and a containment decision due in the next thirty seconds.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.