The AgentOps maturity ladder

The AgentOps maturity ladder

1. Optimized — cost engineering, continuous calibration, an incident flywheel

   The gates hold, so the work moves to making them smarter. Cost per successfully completed task is a tracked metric with a named owner who has actually vetoed a model change. Your judges and thresholds get re-scored against fresh human labels on a schedule, because judge agreement drifts as the agent and the traffic change. And every incident leaves a durable artifact — a new eval case, a new guardrail, a tightened tool scope — with a tracker showing whether it landed.

   Diagnostic question: point at three cases in your golden set and name the incident that created each one. If you cannot, you have a postmortem template, not a flywheel.

2. Managed — CI eval gates, rollout discipline, a kill switch that has been drilled

   The difference between this tier and the one below is enforcement. The eval suite is a gate: a version cannot reach production without passing it, and a named human signs the score. Rollouts are staged — shadow, canary, percentage — with automatic rollback on the metrics you agreed in advance. Every agent has a named on-call owner. New tools and MCP servers go through an allowlist with an owner and a review date.

   Diagnostic question: can the on-call engineer stop this agent — all runs, or just one dangerous tool — in under five minutes, without shipping a deploy, and has anyone actually tried it this quarter? An undrilled kill switch is a config flag with good intentions.

3. Repeatable — versioned artifacts, retained traces, a golden set someone runs

   You can answer what changed and what did the model see. The prompt, tool schemas, and a pinned model snapshot ship together as one versioned artifact. Traces are captured and retained long enough to investigate — a run from last Tuesday can be reconstructed. A golden set exists, owned by someone who can say an answer is wrong, and gets run before releases.

   The honest weakness of this tier: the evals run when someone remembers. Nothing stops a release that skipped them, so under deadline pressure, releases skip them.

   Diagnostic question: if the person who built the agent left tomorrow, could someone else change its prompt safely and know whether they broke it?

4. Ad-hoc — agents in notebooks

   The agent works — in a notebook, on the builder’s laptop, in a demo. The prompt lives in a Slack thread or a cell that has been edited in place a dozen times. Nobody can say which model version served last week’s runs. A failure is investigated by running it again and hoping.

   This tier is not shameful; it is the correct place to start. It becomes dangerous the moment the agent gets a tool with real side effects, because the signature failure of ad-hoc is you cannot tell whether it is working, and non-determinism means the demo proves nothing about the distribution.

   Diagnostic question: what is the task failure rate? If the answer is a feeling rather than a number, you are here.