Postmortems that feed the machine

Lesson 5 of 5 in Monitoring and Incident Response: When the Agent Is the Incident.

A blameless timeline, contributing causes, and a list of action items is table stakes. For agents there is one additional discipline, and it is the difference between a team that gets better and a team that keeps having the same incident with different words:

Every finding must become an executable artifact. Not a document, not a reminder, not a bullet in a wiki page nobody opens. There are exactly four artifacts an agent incident can produce, and if a finding does not map to one of them, it has not been fixed.

The postmortem flywheel

  1. Incident closed

    Contained, diagnosed, service restored in stages. Nothing durable has been gained yet.

  2. Postmortem: timeline, contributing causes, blast radius

    Blameless, and specific about what evidence was missing — instrumentation gaps are findings, not excuses.

  3. Route each finding to an artifact

    One primary artifact per finding. A finding with no artifact is a wish.

  4. Eval case — the failing run becomes a test

    The exact task, inputs and expected outcome, added to the suite that gates releases. Write it so it fails against today’s build before you fix anything.

  5. Guardrail entry — a runtime check that blocks the class

    Argument validation, output filter, injection check on tool output, destructive-operation block. A guardrail blocks the whole class, not the instance you saw.

  6. Permission change — remove the capability

    Narrow the credential, drop the tool, split the environment. The strongest artifact: a capability that does not exist cannot be misused, injected into, or looped over.

  7. Detection or instrumentation gap — a signal that should have fired

    A new alarm from lesson two, a missing span attribute, a version stamp nobody recorded, or retention that expired before you looked.

  8. Merge into the release gate

    Eval cases enter the CI suite; guardrails and permissions ship as config with tests; alarms get thresholds and an owner. Artifacts that live outside the pipeline decay within a quarter.

  9. The next release cannot silently regress this failure

    That is the whole point of the flywheel: your agent gets measurably harder to break with every incident you have.

The four artifacts are not equally strong, and the ranking matters when you are trading effort. Removing a capability beats blocking its misuse at runtime, which beats testing that it does not happen, which beats noticing when it does. Reach for the strongest artifact the product can tolerate, then add the weaker ones underneath as defence in depth.

Sort the findings below. Each one is written to point at a single primary artifact — the skill is hearing which.

Interactive sorting exercise: Eight real postmortem findings. Which artifact does each one become? (Most findings deserve more than one; pick the primary — the artifact without which the fix does not hold.)

Interactive flashcard deck.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.