Monitoring that actually catches these
Lesson 2 of 5 in Monitoring and Incident Response: When the Agent Is the Incident.
A dashboard is not monitoring. Monitoring is the set of conditions that will wake someone up, and for agents that set is small, specific, and almost entirely absent from default infrastructure alarms. Latency, CPU and 5xx rates tell you the harness is healthy. They say nothing about whether the agent spent £4,000 looping on one ticket.
Here is the alarm set, mapped directly onto the catalogue from lesson one. Build these seven and you have coverage of all eight failure modes; skip them and you are relying on customers as your detection layer.
| Alarm | Fires on | Catches (catalogue #) | Page or ticket? | The trap |
|---|---|---|---|---|
Loop-depth alarm | Turns-per-run p99 crossing ~80% of the cap, or three identical (tool + normalised args) calls in one run | 1 infinite loops, 2 thrash | Page if it fires across runs; ticket for a single run | Alarming on the cap itself is too late — by then you have paid for the whole run. Alarm on the approach. |
Cost-rate alarm | Spend per minute against a rolling baseline, plus per-tenant spend anomaly and cost per successful outcome | 7 runaway cost, and 1/2 upstream of it | Page — money is the one signal that never self-heals | Cumulative monthly-spend alerts fire after the loss. Rate beats total; per-tenant beats aggregate. |
Guardrail-trip spike | Rate of guardrail blocks per 1,000 runs — injection patterns in tool output, argument allowlist denials, egress denials, output filters | 3 hallucinated args, 5 context poisoning | Page on a step change; ticket on drift | A guardrail-trip rate of zero is not good news — it usually means the detector broke or stopped being called. |
Success-rate drop | Task success (online judge, downstream confirmation, or human accept) versus the last stable release, per task type | All eight — it is the roll-up | Page on a sharp break; ticket on slow decay | Retries and fallbacks hide it: a run that succeeds on attempt four is a success in your metric and a disaster in your cost and latency. |
Tool error and schema-failure rate | Per-tool 4xx/5xx, argument schema-validation failures, not-found rate, and your own fuzzy-match counter | 3 hallucinated arguments, 2 thrash | Ticket, unless a single tool breaks outright | Tools that "helpfully" coerce bad arguments report success and delete this signal entirely. |
Truncation and context-utilisation | Harness truncation counter, bytes-returned versus bytes-in-context delta, context utilisation above ~90% on any turn | 4 silent truncation, 8 degradation | Ticket — but with a hard error at the step for decision-critical tools | This is the only failure mode in the list that produces zero anomalies in latency, cost or error rate. If you do not instrument it, you will never see it. |
Human-override and escalation rate | Rejections at approval gates, median review time, edit-before-send rate, escalation-to-human rate | 6 stale memory, 8 degradation, plus autonomy drift | Ticket, reviewed weekly | Falling rejection rates read as "the agent got better" and just as often mean reviewers stopped reading. Track review time beside it. |
Two rules make this set work in practice.
Set thresholds from your own baseline, not from a blog post. There is no universal correct value for turns-per-run or cost-per-task; there is only your distribution over the last two weeks and a multiple of it. Measure for a fortnight, set the alarm at a level that would have fired two or three times, then tighten. A threshold you copied is a threshold nobody owns.
Alarm per tenant and per agent version, not globally. Non-deterministic systems have fat tails, and one pathological tenant hides comfortably inside a healthy aggregate. Every alarm above should carry a dimension for tenant, task type, and the agent version tuple — which is also exactly what makes the forensics in lesson four possible.
From the harness
Sees: every loop iteration, the turn counter, the stopping reason, truncation events, retries, which tool the runtime allowed, gated or denied, and the resolved version tuple. The harness is the only vantage point that knows the difference between requested and executed.
Cannot see: whether the outcome was any good.
Alarm from it: loop depth, repeated-call counter, truncation, guardrail trips, gate denials. If your harness emits no counters, no amount of platform tooling will reconstruct them — this instrumentation is your job, not your vendor's.
From the model gateway or proxy
Sees: every model call — tokens in and out, cost, latency, model id, and which credential made the call. It sits at a chokepoint, which makes it the cheapest place to enforce as well as observe.
Cannot see: the agent's intent, or whether two calls belong to the same run unless you propagate a run id through headers.
Alarm from it: cost rate, token-per-task, latency, per-credential anomalies. Enforce from it too: a per-run and per-tenant budget cut-off at the gateway is the containment you will be grateful for at 03:00.
From the trace backend
Sees: the whole run as a span tree — plan, tool calls with arguments, retrieved chunks, outputs, timings — and, once you derive metrics from spans, the ratios that matter: calls per successful outcome, quality by turn index, tool error rate by task type.
Cannot see: anything you did not put on a span. Traces are not free telemetry; they are a recording contract you wrote earlier.
Alarm from it: success rate, calls-per-outcome, quality-by-turn-index, and any online judge scores. This is also your only forensic surface — see the trace-anatomy crosswalk for what each span must carry.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.