What to record on every span

Lesson 2 of 5 in Trace Anatomy: Reading an Agent Run Like a Professional.

Instrumentation is a design decision, not a checkbox, and there is one rule that generates the whole field list:

Record what you would need to answer “why did it do that?” without re-running the agent.

Agent runs decompose into three families of span, and each family answers that question with different fields. Model calls explain choices. Tool calls explain observations. Decision and handoff spans explain routing — the branch the harness took, and on what basis.

Model call spans

Record:

  • Requested model and served model. Not the same field. Gateways, aliases, fallbacks and capacity routing can serve something other than what you asked for; if you only log the request, a quality shift is unexplainable.
  • Input and output tokens, separately, plus cache-read tokens where the provider reports them. Per-call, not just per-run — the run total is derived, the per-call number is the evidence.
  • Finish reason. tool_use, end_turn, max_tokens, stop_sequence, content-filter — this single enum separates “the model finished its thought” from “the model ran out of room mid-sentence”.
  • Sampling config: temperature, top-p, seed if you set one, and the tool-choice mode.
  • Latency, and for streamed calls time to first token — a p95 latency chart without TTFT hides the difference between a slow model and a slow first byte.
  • A reference to the messages, not necessarily the messages: a pointer into a payload store, plus a hash, keeps the span small and the context window contents governable.

The field teams forget: the finish reason. A max_tokens finish on turn 5 explains a malformed tool call on turn 6 in one glance.

Tool call spans

Record:

  • Tool name and the arguments the model supplied, redacted or hashed where they carry secrets. The arguments are the model’s actual decision — the most diagnostic bytes in the whole trace.
  • Result size in bytes and a truncated flag with the original size. If your harness clips tool output to fit the context, that clipping is a semantic event and must appear in the trace.
  • Duration and status, where status reflects the meaning of the result and not merely the transport code. A 200 response whose body says {"ok": false} is a failed tool call.
  • Error class on failure — timeout, auth, rate-limited, validation, not-found — so you can aggregate. Free-text error strings do not aggregate.
  • Retry count and the downstream request id, which is how you hand the problem to the team that owns the dependency.
  • Whether the call was gated, by what rule, and what the human decided. An approval gate with no span cannot be audited.

The field teams forget: the truncation flag. It is the single highest-yield attribute in this list, and lesson four is mostly about what happens without it.

Decision and handoff spans

Not every span is an I/O call. The harness makes choices too, and they are invisible unless you record them:

  • Which branch the router took, on what criterion, and what the alternatives were. “Routed to billing_agent because the classifier scored 0.81 against a 0.7 threshold” is a debuggable sentence; a silent handoff is not.
  • Which subagent received the handoff, with its own child span carrying its own token counts. This is how you attribute cost and blame in a multi-agent system.
  • Guardrail and validation outcomes: which check ran, pass or fail, on which content, and what the harness did about it. A blocked output that leaves no span looks exactly like a model that produced nothing.
  • Budget state: turns used of turns allowed, tokens spent of the cap. A run that ended at turn 14 of 14 is a different story from one that ended at turn 6 of 20.
  • Context management events: compaction, summarisation, message eviction. If the harness silently rewrote history, that rewrite is the reason later turns "forgot" something.

The field teams forget: compaction. An agent that loses a constraint mid-run is usually an agent whose harness summarised it away.

Then the counterweight. A trace store is queried by engineers, support, data scientists, and — after an incident — auditors, and it retains months of every payload your agent ever touched. It is simultaneously your most useful and your most dangerous datastore.

So put the skeleton in the span and the blobs behind a reference. Sizes, hashes, classes, counts and enums go in attributes: they are cheap, they aggregate, and they leak nothing. Raw payloads go to a payload store with its own retention and access control, addressed by a pointer on the span. Redact at the SDK boundary, before the exporter, because anything that reaches the collector is already replicated.

Interactive sorting exercise: Ten fields from a real instrumentation review. Where does each belong?

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.