Watching agents: tool failures and truncated streams

Lesson 3 of 4 in Monitoring Models and Agents in Production.

A web service fails loudly — a 500, a timeout, a red graph. An agent can fail plausibly: it returns a confident paragraph after its tool call quietly errored, hands off to the wrong sub-agent, or stops streaming mid-sentence with a 200 status. The MLA-C02 guide names three agent failure classes to monitor: coordination failures between agents and tools, truncated streaming responses, and tool invocation failures. Each has a distinct telemetry signature — and none shows up on a CPU chart.

Coordination failures — the agents stop agreeing

A planner loops on the same step, a hand-off sends a billing question to the search agent, context goes stale between steps. Log: a trace of every step — which agent/tool was chosen, with what input, and why (session id stitching the whole run together). Alarm on: steps-per-session and session duration outliers (loops), hand-off target distribution shifts, and sessions that end without reaching a terminal state.

Truncated streaming responses — the answer just stops

The stream ends mid-answer: the model hit its max output-token limit, a timeout or network drop cut the connection, or an intermediary buffered the stream to death. Often there is no error status at all — the user simply sees half an answer. Log: the stop reason and output token count for every response. Alarm on: a spike in max-token stop reasons or an abnormal share of unusually short responses.

Tool invocation failures — the hands stop working

The model emits arguments that fail the tool's schema, the tool's Lambda throws or times out, or the agent's role lacks permission to call it. The agent may retry, hallucinate the result, or apologize vaguely. Log: per-call tool name, arguments, latency, and error class. Alarm on: per-tool error rate and latency — one broken tool poisons every workflow that routes through it.

Where does this telemetry come from? For agents on Bedrock, AgentCore Observability emits built-in metrics for each AgentCore primitive — runtime, memory, gateway, built-in tools, identity — and session-level traces of every step and tool call, viewable on the CloudWatch GenAI observability dashboard's AgentCore tab (self-hosted agents join via the ADOT SDK). The next module treats it as a troubleshooting tool; here the point is what to record: step traces, stop reasons, and per-tool outcomes.

One agent turn, instrumented

  1. User request

    Session id assigned — every later event carries it.

  2. Agent reasons / picks tool

    Trace records the decision and its inputs.

  3. Tool call succeeds?

    Log tool name, arguments, latency, error class.

  4. Observe result, continue

    Loop guard: alarm when steps-per-session goes outlier.

  5. Stream final answer

    Record stop reason + output tokens — the truncation tell.

  6. Complete (terminal state)
  7. Retry / fallback + alert

    Per-tool error-rate alarm catches systematic breakage.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.