The first three things: traces, a golden set, a kill switch
Lesson 5 of 5 in AgentOps Org Patterns: Maturity, Team Shapes, and Gates That Scale.
A team adopting agents asks the wrong first question — which platform should we buy? — because it is the easiest question to answer. Here is the right one: what three capabilities do we need before we let an agent touch anything that matters?
Traces. A golden set. A kill switch. Everything else in this domain — CI gates, canary rollouts, judge calibration, cost engineering, the incident flywheel — is built on top of those three. None of them is a purchase, all three have an embarrassingly cheap first version, and each one makes a whole class of work possible that is simply impossible without it.
| Capability | Honest minimum version | What it unlocks | What is impossible without it |
|---|---|---|---|
1 · Traces | Structured records — one per run — of the inputs, every model call, every tool call and its result, the final action, and the version ids. JSON lines into the log store you already own, a searchable run id, a retention window you can defend, PII handled deliberately. | Reconstructing what the model saw. Harvesting real failures into eval cases. Measuring cost and latency per task. Any postmortem worth the meeting. | Debugging a non-reproducible failure. Any honest statement of a failure rate. Every other capability on this list. |
2 · A golden set | 20–50 real cases with expected outcomes, in a file in the repo, owned by a named domain expert. At least half harvested from real traces — including the two failures that scared you. Run by a script, not by hand. | Answering “did this change help?” with a number. The CI eval gate. Judge calibration. Comparing models or prompts without arguing. | Shipping a change on evidence rather than a demo. Distinguishing a bad release from a bad day. |
3 · A kill switch | A flag the runtime checks per run and per tool call, with scopes for one agent, one tool, and one tenant — flippable from a runbook by whoever is on call, without a deploy. Plus one drill on the calendar. | Containment as the first incident move. Raising the autonomy dial at all. Sleeping through a canary. | Any honest claim of containment. Otherwise every incident starts as an argument about whether to roll back, while the agent keeps acting. |
What should this team stand up next?
Interactive decision tree — outcomes:
- Kill switch first — today
An agent taking irreversible actions that on-call cannot stop is the one configuration with no defensible version. Ship the flag the runtime checks per run and per tool call, give on-call standing authority to flip it, and write the runbook line. Everything else waits a week.
- Drill the switch you already have
Untested containment is the most common false claim in agent operations. Run the drill in a real environment: flip it, watch an in-flight run stop, time it, and note what surprised you — usually that it only takes effect between runs, or that the person with permission was on holiday. One hour of work converts a hope into a capability.
- Traces next
Final-output logging cannot answer the only question that matters in an agent postmortem: what did the model see when it decided? Emit one structured record per run — inputs, every model and tool call, results, final action, version ids — into the log store you already have. This is a week of work and it unlocks both of the other capabilities.
- Golden set next — harvested, not imagined
Now that you have traces, pull 20–50 real cases into a file in the repo, weighted towards the failures that actually happened, and give it a named domain-expert owner. Run it with a script. You have just replaced “it worked in the demo” with a number, which is the whole repeatable tier.
- Your golden set needs an owner and real cases
Hundreds of unread synthetic cases are worse than twenty real ones: they pass, they inflate confidence, and nobody can say what a failure means. Have the domain expert read a sample, delete what does not reflect real traffic, and replace it with harvested traces — including the failures that embarrassed you.
- Turn the suite into a gate
You are at the repeatable tier with everything needed for managed. Wire the suite into CI so a failing version cannot ship, agree the threshold before the run, and name a signer who records what they would have blocked on. Then add staged rollout with automatic rollback, and you have the managed tier in full.
- You are managed — go build the flywheel
Traces, an owned golden set, an enforced gate, and drilled containment. The next tier is not more mechanism: it is cost per successful task with an owner who can veto, judge thresholds re-calibrated against human labels on a schedule, and every postmortem leaving a merged eval case. Check the flywheel the honest way — name the incident behind three of your eval cases.
Interactive flashcard deck.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.