From traces to evals, cost, and forensics
Lesson 5 of 5 in Trace Anatomy: Reading an Agent Run Like a Professional.
One artifact, four consumers — and this is why trace design is worth an afternoon of argument rather than a default config.
Evals consume traces as their unit of judgement. Outcome scoring asks whether the final answer was right; trajectory scoring asks whether the path was reasonable — did it pick the right tool, pass the right arguments, use the result it got? Every one of those questions is a query over span attributes, which is why Amazon Bedrock AgentCore Evaluations takes OpenTelemetry traces as its input and scores agent interactions from them with an LLM judge. Your span quality sets a ceiling on your eval quality.
Cost accounting needs per-model-span token counts, the served model id, and the grouping keys on the root span. With those you can answer “what does this feature cost per successful run, per tenant, this week?” Without them you have a provider invoice and a shrug.
Incident forensics needs the trace to survive as evidence: immutable, retained long enough for the investigation, redacted enough to be safe, and access-controlled enough to be shareable. And product analytics wants the boring aggregates — task success rate, turns per task, escalation rate — which are just group-bys over the same rows.
| Consumer | Depends on | Breaks without it |
|---|---|---|
Offline evals on a golden dataset | Tool name and arguments, result status and size, ordered span sequence, run stop reason, prompt and agent version on the root. | You can score the final answer but never the trajectory — so a run that got the right answer by luck scores identically to one that reasoned correctly. |
Online monitoring and alerting | Semantic span status, error class, run stop reason, truncation flag, token and latency metrics. | Every drill in the last lesson passes your alerting untouched: green spans, plausible answers, no page. |
Cost and capacity accounting | Per-call input/output/cache tokens, served model id, tenant and feature keys on the root span. | Costs can be totalled but not attributed — no per-tenant margin, no “which turn got expensive”, no case for the optimisation you want to fund. |
Incident forensics | Complete unsampled traces for affected runs, payload pointers, gate and approval spans, immutability and a known retention window. | You cannot reconstruct what the agent did or prove a human was in the path — and the run will not reproduce on request. |
Which sets up the tension every team hits in month three: traces are simultaneously the most useful and the most expensive thing you store, and the most sensitive. Full-fidelity traces with payloads for every run will outgrow your agent’s own compute bill and will contain, somewhere in them, every customer secret your tools have ever touched.
Sampling is the usual answer, and naive sampling is a trap. Keep 1% of runs at random and you have thrown away 99% of the rare failures — which are the entire point. Sample on decisions, not on dice: keep everything interesting, keep a baseline slice of the boring, and keep the skeleton even where you drop the payloads.
How much of this run do you keep?
Interactive decision tree — outcomes:
- Retention is not your problem — the agent is
If a quarter of your runs end in a budget, an error or an escalation, “keep the interesting ones” has stopped being a sampling policy and become “keep everything”. Read a handful of them properly first: at that rate you are almost certainly looking at one systematic failure — a loop that never converges, a tool that fails semantically, a truncation limit set too low. Fix that, and the tail shrinks back to a tail.
- Full trace, payloads behind a guarded store
Keep spans and attributes in the trace backend; keep raw payloads in a separate store with its own access control, shorter retention, and an audit log of reads. Mask at the SDK boundary first — anything that reaches the collector is already replicated. This is the tier your incident responders and auditors need, and the tier your privacy review will actually ask about.
- Full trace with payloads
Interesting and not sensitive: keep everything, and keep it long enough to cover a slow investigation — a quarter is a common floor. These are the runs your evals will mine for regression cases and your engineers will read line by line.
- Skeleton only — spans, attributes, no payloads
Names, timings, statuses, token counts, sizes, hashes and grouping keys are cheap, aggregate beautifully, and leak nothing. Almost every population question you will ask is answerable from the skeleton alone, which makes this the right default for routine successful runs.
- Skeleton for all, full payloads for a small random slice
Keep a few percent of ordinary runs at full fidelity so you have a picture of healthy behaviour to compare failures against — and so a judge or a new engineer can read what “normal” looks like. Random within the boring population is fine; random across all runs is the trap.
- Drop it — but be honest about what you lose
Dropping ordinary runs entirely is defensible only when you have no need to compare populations and no obligation to reconstruct behaviour. In practice teams that drop by default discover the gap during their first incident, when the interesting run turns out to have looked ordinary until someone asked a question about last Tuesday.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.