The five charts an agent team actually looks at

1 · Task success rate, segmented by cohort — outcome-judged, split by tenant, task type, model version and prompt version

“Is the agent doing the job, and for whom is it not?”

Below threshold on a cohort → auto-ticket with the cohort filter and exemplar traces attached; investigate that segment, add its failures to the golden dataset.

Unsegmented, it moves whenever your traffic mix moves and you chase phantom regressions. Without model and prompt version stamped on it, you cannot tell a bad release from a provider rollover.

2 · Cost per successful task — total spend ÷ successful runs, not spend ÷ runs

“What does one unit of delivered value cost, and is that trending the wrong way?”

Rising → look at steps per run and retry rate before you touch model choice; the usual cause is failures burning tokens, not inference prices. Sustained rise → budget review or a cheaper model on the easy cohort.

Spend per run falls when the agent gives up early — a metric that improves as quality collapses. Dividing by successes is what makes it honest.

3 · Latency percentiles with a model-versus-tool split — p50, p95, p99, and where the time went

“How slow is the slow tail, and is it our inference or somebody else’s API?”

p99 up with tool time dominating → chase the dependency or add a timeout and a fallback. Up with model time dominating → look at context growth, retries and prompt size.

A mean hides the bimodal shape entirely. A percentile with no model/tool attribution tells you that you are slow and gives you nowhere to go.

4 · Steps per run and turn-limit terminations — the loop depth distribution, plus the count of runs killed by the ceiling

“Are agents thrashing, and how often does the turn limit become the de facto stopping condition?”

Fat right tail → open the deepest traces; the usual causes are a tool returning unusable output, a missing definition of done, or a retry loop. Ceiling hits above a fraction of a percent → treat as a bug, not a budget setting.

A mean step count is nearly useless — thrashing is a tail phenomenon. And counting only successes hides the runs the limit killed, which are precisely the interesting ones.

5 · Intervention and guardrail rate — approvals, escalations, guardrail trips, blocked tool calls, kill switch pulls, plus median review time

“Are the humans still meaningfully in the loop, and is anything hostile happening?”

Guardrail trips spiking → security triage as a possible prompt injection campaign. Approval rate approaching 100% with review time approaching zero → the gate has hollowed out; re-scope what needs approving.

A raw approval count says nothing. Approval rate next to review time is the pair that exposes rubber-stamping — the single most common oversight failure in deployed agents.