The agent failure-mode catalogue

Lesson 1 of 5 in Monitoring and Incident Response: When the Agent Is the Incident.

Traditional services fail in ways your monitoring already knows: the process died, the disk filled, latency went up, the error rate spiked. Agents fail those ways too — and then they fail in eight more ways that look, from the outside, like nothing at all. The CPU is fine. The HTTP 200s keep flowing. The agent is confidently doing the wrong thing at full speed.

This lesson is a reference block, not a story. Learn it once and you will recognise every agent incident you ever get paged for as one of these eight shapes, or a chain of two of them.

Read each entry in three parts, because that is the order you need them at 03:00:

  • Symptom — what a human notices first. Usually a bill, a complaint, or a weirdly quiet dashboard.
  • Detection signal — the metric or query that would have caught it before the human did. This is what you build in lesson two.
  • First response — the action you take in the first five minutes. It is almost never "read the logs"; it is almost always narrow the agent's authority, then read the logs.

1 · Infinite loops — the run that never converges

Symptom. A run hits its turn cap. The trace shows the same search repeated with near-identical arguments, or a two-step cycle (read file → plan → read the same file → plan). No terminal action ever happens. Cost per task quietly triples.

Detection signal. Turns-per-run at p95/p99 approaching the cap; a repeated-call counter that hashes (tool name + normalised arguments) and fires at three or more identical calls in one run; time-to-first-side-effecting-action.

First response. Kill the run — do not let it "finish" — and keep the trace. Then check whether a stopping condition is actually defined and observable to the model. A turn cap that fires often is not a safety net working; it is your only stopping condition, which means the agent has none.

2 · Tool thrash — motion without progress

Symptom. The agent oscillates: creates a record, deletes it, recreates it; switches between two overlapping tools; re-reads what it already has. Every individual call succeeds, so error-rate dashboards stay green.

Detection signal. Tool calls per successful outcome — the single most diagnostic agent metric there is. Track it per task type and alarm on the ratio, not the count. Also: writes-then-inverse-writes on the same resource id, and repeated reads of the same resource within one run.

First response. Rate-limit the thrashing tool for that agent version and freeze writes to the affected resource. Thrash usually traces back to two tools with overlapping descriptions, or a tool whose response does not tell the model that the action already succeeded.

3 · Hallucinated arguments — plausible ids that do not exist

Symptom. The agent calls a real tool with invented arguments: an order id in the right format for a record that never existed, a file path that looks like your repo, a customer email assembled from a pattern. Downstream you see a spike in not found responses — or, far worse, nothing, because something fuzzy-matched.

Detection signal. Schema-validation failure rate per tool; 404/not-found rate per tool; guardrail trips on argument allowlists; and — the one people miss — fuzzy-match rate inside your own tool implementations.

First response. Make the tool reject rather than approximate: exact match or a structured error, never a best guess. Return the error to the model as data it can act on, and stop any blind auto-retry loop. If an invented argument could have selected a real record belonging to someone else, this stops being a reliability bug and becomes an access-control incident.

4 · Silent truncation — the agent read half the document

Symptom. A confident, well-written, wrong answer. The agent summarises a 200-page contract and never mentions the termination clause on page 180. Nothing failed. Nothing retried. The output looks like the good outputs.

Detection signal. Bytes returned by the tool versus bytes actually placed in the context window, per span — a delta means truncation. An explicit truncation counter emitted by the harness. Context utilisation above ~90% on any turn. Watch for the tell-tale pattern of quality dropping only on large inputs.

First response. Turn the silence into an error: the harness must insert a visible marker (truncated 180 of 200 pages) and, for anything decision-critical, fail the step rather than continue. Then paginate the tool. Re-run the affected tasks — truncation produces outputs that are silently wrong, not runs that are visibly failed, so nothing self-heals.

5 · Context poisoning — the agent read instructions it should have read as data

Symptom. Behaviour changes mid-run, right after the agent ingested external content: a web page, a ticket comment, a retrieved document, a tool response. It starts pursuing a goal nobody set. Sometimes benign (a stale runbook telling it to do the old thing); sometimes indirect prompt injection or poisoned tool output.

Detection signal. Guardrail trips on instruction-like patterns in tool output (not just user input); divergence between the plan the agent stated and the actions it then took; egress to a domain never seen before for that agent; a jump in the entropy of the action distribution for a task type.

First response. Contain before you read. Revoke egress and credentials, then kill the run — in that order — because a poisoned run may be actively exfiltrating while you scroll the trace. Quarantine the source document so the next run does not re-ingest it. Then hand off to the security incident path; this is an attack until proven otherwise.

6 · Stale memory — yesterday’s wrong answer, forever

Symptom. The agent repeats one specific mistake across sessions, cites a policy that changed last quarter, or "remembers" a fact about a customer that was corrected weeks ago. Complaints cluster oddly: same wrong answer, different users.

Detection signal. Age distribution of memory entries actually read per run; share of answers grounded in memory rather than a live source of truth; automated contradiction checks between memory entries and the system of record; clustered thumbs-down on one narrow topic.

First response. Snapshot first, then invalidate. Scope the invalidation to the affected entries and pin the agent to live retrieval for that topic. Wiping memory feels decisive and destroys the only evidence of what the agent believed — and belief is the thing you are investigating. Longer term: every memory write needs provenance and a TTL.

7 · Runaway cost — the bill as a failure detector

Symptom. Spend goes vertical. Almost always a symptom of another entry in this list — loops, thrash, or a retry storm — surfacing on the one dashboard finance watches. On a bad day it is a single tenant with a pathological input; on a worse day it is every tenant at once after a prompt change.

Detection signal. Cost rate (spend per minute against a rolling baseline), not cumulative spend, which alarms only after the money is gone. Also tokens per task, cost per successful outcome, and per-tenant cost anomalies so one customer cannot hide inside the aggregate.

First response. Enforce a hard budget cut-off at the gateway — per run and per tenant — and let runs fail closed. Then find the upstream failure mode. Raising the limit to "unblock the customer" converts a bounded reliability incident into an unbounded one.

8 · Degradation over long runs — good at five turns, bad at forty

Symptom. Quality decays with run length. Early instructions get ignored, the output format drifts, the agent forgets a constraint it honoured on turn three, and it starts re-deriving conclusions it already reached. Short evals pass; long production runs disappoint.

Detection signal. Outcome quality bucketed by turn index — the chart most teams never draw. Instruction-adherence checks sampled late in the run. Context utilisation plotted against success rate; the knee in that curve is your real turn budget.

First response. Cut the maximum turn count to just past the knee and split the task, rather than letting long runs limp on. Checkpoint and restart with a compacted context instead of accumulating one. Treat "works short, fails long" as an architecture finding, not a model deficiency — bigger context windows move the knee, they do not remove it.

Interactive sorting exercise: Ten symptoms as they actually arrive — a bill, a complaint, a graph. Name the failure mode. (Chains are real; sort by the mode that is doing the damage.)

Key terms: trace, span, observability, stopping condition, tool-output poisoning, memory

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.