Three signals: notice, explain, prove
Lesson 1 of 5 in Observability in Production: Watching Agents at Scale.
You already know how to read one trace. Production breaks that skill in a specific way: at ten thousand runs a day, nobody reads traces. They get read after something else told you which run to open.
That “something else” is the whole discipline. Production observability for agents is three signals with three different jobs, and teams that collapse them into one — usually “we log everything to the trace viewer” — end up with a system that can explain any failure they already know about and notice none of the ones they do not.
| Signal | Unit | Cost shape | The question it answers | What breaks if it is your only signal |
|---|---|---|---|---|
Metrics — aggregates | A number per time bucket, sliced by a few low-cardinality labels (agent, version, tool, tenant tier). | Nearly flat in traffic. Cost grows with label cardinality, not run count. | “Is something different today?” | You detect the incident and cannot explain it. A success rate that fell from 91% to 78% with no traces behind it tells you to panic, not what to fix. |
Traces — per-run records | One trace per run, spans for each model call, tool call, retrieval and handoff. | Linear in traffic and in run length. The single largest line item in most agent observability bills. | “What exactly happened in this run?” | You can reconstruct any run you think to look at, and you never think to look. Nothing in a trace viewer tells you that yesterday was 4% worse than last week. |
Logs — durable text | Structured events with run id and span id attached, retained on a compliance schedule and redacted at emit. | Linear in traffic, but retained far longer than traces — which is where the cost and the risk both live. | “What did we do, for whom, and can we show it?” | You have an archive nobody queries. Logs answer questions asked in a legal or security register, not an engineering one — they are the wrong tool for a Tuesday afternoon regression. |
The metrics list is where agents genuinely differ from the web services your monitoring stack was built for. HTTP 200 means the request was served; it says nothing about whether the agent did the job. So the aggregate layer needs agent-shaped numbers:
Task success rate — the fraction of runs that achieved the goal, judged by an outcome check, not a status code. Token spend per run, split model-by-model, because a retry storm is invisible in latency and obvious in tokens. Latency percentiles — p50, p95, p99, never the mean. Loop depth, the number of iterations before the stopping condition fired, because a rising tail is the earliest signal of an agent that has started thrashing. Tool error rate per tool, since one flaky dependency degrades the whole run in ways the model will paper over. And intervention rate — approvals, escalations, guardrail trips, kill switch pulls — which is the only number that tells you whether your humans are still in the loop or have become a rubber stamp.
Interactive sorting exercise: For each production question, which signal answers it *first*? Choose the cheapest signal that can actually answer — the others are for the next step.
Key terms: observability, trace, span, redaction, loop depth, kill switch
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.