One request, one story
Lesson 5 of 5 in Orchestration End to End: One Request Through the Whole System.
Everything above is design. This is the payoff: if the ids from lesson three were threaded properly, the entire journey — router, three warehouse calls, a subagent with its own internal loop, a four-hour human suspend, two side effects — reads as one trace, and the seams become visible instead of theoretical.
The requirement is exact and frequently missed: one trace id, propagated into subagents and out to tool servers. If your delegated research task emits its own separate trace, you do not have a system you can debug; you have two stories and a guess about how they relate.
Here is the trace for our request, collapsed to the hop level, with wall-clock at the left. Read it before reading the analysis under it.
Three readings of that trace, each answering a different question.
Where did the time go? The critical path is classify → resolve → subagent → merge → propose → execute → reply. The entitlement subagent is 5.94 of 11.28 seconds — 53% of the run. The three warehouse calls contributed zero to the critical path, because they ran in parallel with a branch that took six times longer. Note what that means for the obvious optimisation: the team’s standing complaint is warehouse 2’s ERP, and making warehouse 2 instant would save nothing at all. The wins available are inside the subagent — fewer pages read, a smaller model for turn 1, or a cache.
Where did the money go? 38,400 of 86,420 tokens — 44% — went to one subagent re-reading the same 41 pages of a contract that changes roughly never. That is not a model problem or a prompt problem. It is a caching problem at a seam, and it is invisible unless tokens are attributed per hop. Extract the entitlement per contract once, store it with a provenance record and an expiry, and this hop becomes a lookup.
What actually happened to the world? Two effects, both keyed, both recorded with their downstream ids (SH-77120, CM-3391), and one human decision that changed the parameters — the approver raised the quantity from 4 to 6. That last detail is why approval belongs in the trace: the executed remedy is not the proposed remedy, and six months from now that difference is the whole question.
| Attribute | Answers |
|---|---|
Hop name + control owner | Was this hop’s next step chosen by code or by the model? The control ledger, verifiable after the fact instead of aspirational |
Exposed tool set | Could the model even see the right tool? The one log line that makes selection bugs falsifiable — see the tool-selection module |
Model, tokens in/out, cost | Which hop spends the money. Aggregate to cost per resolved task, not cost per call |
Budget remaining on entry | Whether a timeout was a slow dependency or a deadline that was already nearly spent when the hop began |
Coverage / partial flags | Whether a downstream answer was built on complete inputs — the field that stops a gap being invisible |
Idempotency key + downstream id | What changed in the world, and whether a retry would repeat it |
Human decisions, with edits | Who approved what — including where the approved parameters differ from the proposed ones |
Tool: Orchestration Sandbox — Wire hops together and watch the seams misbehave — partial results, budget exhaustion, and control-flow ownership, in a system you can break on purpose.
Tool: Trace Debugger — Then practise the discipline above on traces that fail in each of the four candidate ways.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.