The retrospective: trace, logs, evals, risks
Lesson 5 of 5 in Anatomy of a Real Agent: One Run, Every Part Named.
Step back from the run. What you have been reading for four lessons has a name: a trace — the complete, ordered record of one agent run. Each timed unit of work inside it is a span: the model generation in turn 1 is a span, the 218 ms lookup_order execution is a span nested under it, the failed carrier_track call is a span whose status is error. Turns group spans; the trace is the whole tree. This vocabulary comes from distributed-systems observability, and the agent world adopted it because an agent run is a distributed system: one probabilistic component and several deterministic ones, passing messages.
Two records of this run now exist, and confusing them is a classic beginner mistake. The transcript is what the model saw — prompts, calls, results. The trace is what the system recorded — all of that plus what no model ever saw: latencies, gate decisions, budget counters, schema-validation results. The transcript is inside the trace; the trace is strictly bigger. When a run goes wrong at 2 a.m., the trace is the only witness — which is why "what would we log?" is a design-time question, not a post-incident one.
First, prove you can read one. Tag each excerpt below with its stage in the loop.
Interactive sorting exercise: Drag each trace excerpt to the loop stage it belongs to.
| Segment | What you would log | What you would eval | Where the risk lived |
|---|---|---|---|
Setup | System-prompt version, tool registry, gate config, budgets | Prompt-regression suite: does policy wording still produce compliant behaviour after edits? | Policy stated in prose but not enforced in code — the prompt-deep-control trap |
Turns 1–2 | Full tool calls + arguments, reasoning summaries, per-span latency | Turn-level: right first tool chosen? Arguments drawn from the right fields? | The wrong-field bug — schema-valid, semantically wrong, invisible until execution |
Turn 3 | The error verbatim, retry count, budget consumed | Recovery rate on a golden dataset of seeded tool failures; loop detection | Uninformative errors → invented statuses; poisoned errors → injection via the error channel |
Turns 4–5 | Gate input, decision, and threshold version — the audit trail for the money | Gate correctness under fuzzed amounts; refund-policy compliance across scenarios | The irreversible action: one probabilistic step from real dollars, held by one deterministic check |
Turn 6 | Final message, total calls/tokens/latency, stop reason (natural vs budget vs halt) | Run-level: issue resolved? Numbers grounded in tool results? Tone on-policy (LLM-as-judge) | Fluent wrap-up that misstates what actually happened — the last mile of grounding |
That table is secretly a map of the rest of this site. Read its columns again as domains:
Architectures asks what happens when this run stops being enough — when Fernway wants returns, exchanges, and warranty claims handled too, and one prompt with five tools becomes a supervisor routing to specialists. The anatomy never changes; it nests. Every multi-agent system is loops inside loops, and you can now read each one.
Security asks where attacker-controlled text could have entered: the customer message, the order record’s notes field, the carrier’s detail string. This agent read third-party data and could move money — two legs of the lethal trifecta — and the reason that stayed safe was runtime design: gates that ignore prose, least privilege tools, no egress beyond the five registered endpoints.
Evals asks how you would know this agent is good before customers do. Not one metric but layers: turn-level assertions (right tool, right arguments), run-level outcomes (refund issued when policy says, not when it doesn’t), and judged qualities (tone, groundedness) — scored against a golden dataset of order scenarios including the failure you watched.
Agentops asks what production looks like: these traces flowing into dashboards, stop-reason and recovery-rate metrics, alert thresholds, a kill switch that can halt every run the moment a refund pattern looks wrong. The trace you just read end to end is the atomic unit of all of it.
Key terms: trace, span, eval, golden dataset, LLM-as-judge, tool-output poisoning
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.
Tool: Trace Debugger — Now debug runs that went worse than this one: step through broken traces, find the turn where each run derailed, and name the missing control.