The trace is the primary artifact

Lesson 1 of 5 in Trace Anatomy: Reading an Agent Run Like a Professional.

A deterministic service gives you two ways to understand a failure: read the code, or re-run the request. An agent gives you neither. Non-determinism means the failing path may never recur, and there is no code path to read — the model chose the path at runtime. What is left is the recording.

That recording is the trace. One trace is one run. Inside it, a span is one timed operation: a model call, a tool call, a retrieval, a guardrail check, a whole subagent. Every span carries five things — a name, a start and end time, a status, a parent, and a bag of key/value attributes. The parent link is what turns a pile of log lines into a tree.

One agent run as a span tree — a refund-triage agent, six turns

  1. invoke_agent · refund-triage (root span)

    Run-level attributes live here: session id, tenant, agent version, prompt version, model config, total tokens, total cost, and the terminal stop reason for the whole run. One row per run — this is what you group and compare on.

  2. 01 · model call — route the ticket

    Attributes: requested model, served model, input/output tokens, temperature, finish reason (tool_use), latency, time to first token if streamed.

  3. 02 · execute_tool · lookup_order

    Attributes: tool name, arguments (redacted), duration, status, result size in bytes, truncation flag, retry count, downstream request id.

  4. 03 · model call — read the order
  5. 04 · execute_tool · get_policy

    A retrieval span nests underneath: the query, the index, the number of chunks returned, the scores. Nesting shows the retrieval was part of this tool call, not a separate model decision.

  6. 04.1 · retrieval · policy index
  7. 05 · model call — propose refund $180
  8. 06 · approval gate — amount > $100

    A gate is a span too: who was asked, what they saw, how long they took, what they decided. Without this span an audit cannot prove the human was in the path.

  9. run ends · stop = escalated_to_human

    The run-level stop reason is the highest-value single attribute in the whole trace. Model-declared-done, turn budget exhausted, error, human abort, escalation — they mean completely different things and they all look identical if you only record status=OK.

Read that tree the way a doctor reads an X-ray: shape first, values second. Six turns, two tool calls, one retrieval, one gate — does that shape match the task you asked for? A good trace-reader builds an instinct for the healthy silhouette of each agent they operate, because most failures announce themselves as a wrong shape long before you inspect a single attribute. Fourteen model calls where you expected four. A tool span with no model span after it. A retrieval that returned nothing and no reaction to it.

The root span deserves special respect. It is the one row per run that everything downstream joins on: your eval harness, your cost report, your incident timeline. Anything you will later want to group by — agent version, prompt version, tenant, entry point, model — belongs there, on the root, not buried in the twelfth child span.

Key terms: trace, span, observability, non-determinism, agent loop, token

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.