Time-to-first-token vs time-to-done
Lesson 3 of 5 in Cost and Latency Budgets You Can Defend.
An agent has two latency clocks, and confusing them is how teams ship something that benchmarks beautifully and feels unusable.
Time-to-first-token (TTFT) is how long until the model starts producing output on one call. It is dominated by prefill — the model reading the resent transcript — so it grows as your run grows. It is the number chat UIs are tuned for.
Time-to-done (TTFD) is how long until the task is finished: every turn, every tool call, every retry, end to end. It is roughly the sum of all per-turn latencies, so a ten-turn run does not take one call’s worth of time — it takes ten, plus the tool time in between. TTFD is the number your user actually waits for and your timeouts actually enforce.
Where the wall clock goes in one turn
- Turn begins
The runtime assembles the payload: prefix, transcript so far, the new turn.
- Queue / rate-limit wait
Provider-side admission plus your own concurrency limiter. Usually small, occasionally the whole story during a burst — and invisible unless you instrument it as its own span.
- Prefill — the model reads the resent transcript
Parallel work, but proportional to input size, so it grows across the run. This is the phase a prompt-cache hit can largely skip — the reason caching shows up on the latency budget as well as the bill.
- First output token — the TTFT clock stops
Everything before this point is what a chat UI feels as “responsiveness”. In an agent, most turns are internal and no user is watching this moment at all.
- Decode — one token at a time
Strictly sequential: a 400-token turn at ~50 tokens/second is ~8 seconds no matter how much hardware you buy. Illustrative rate — but the sequentiality is the durable fact. Shorter model turns are therefore a latency lever, not just a cost lever.
- Runtime validates the tool call
Schema check, permission check, approval gate if one applies. Milliseconds — unless the gate needs a human, in which case this step dominates every other number on the page.
- Tool executes
Network, database, shell, another agent. Fully outside the model’s control and often the largest single term. Instrument every tool call as a span or you will optimise the model while a 4-second API call eats your budget.
- Stopping condition met?
Not met means the whole cycle repeats — with a bigger transcript, so a slower prefill than last time.
- Time-to-done clock stops
Put illustrative numbers on that cycle: 1.5 s of prefill on a 21,000-token payload, 8 s to decode a 400-token turn, 2 s of tool execution, a little queueing — call it ~11 s per turn, so ~110 s for the ten-turn run. Two things follow immediately.
First, decode and tool time dominate, and both are sequential. You cannot buy your way out of them with a bigger instance. Second, the per-turn number barely matters compared to the turn count. Shaving 200 ms off TTFT saves 2 s across the run; removing two turns saves 22 s. In an agent, latency work is turn-removal work.
| Lever | Saves | Mechanism | What it risks |
|---|---|---|---|
Fewer turns — tighter stopping condition, sharper definition of done | Both | Turn count multiplies both meters (it is the n in n·P + g·n(n−1)/2 and in the per-turn latency sum) | Unfinished runs. Earn the cap by measuring how many tasks genuinely need the last few turns |
Prompt caching | Both | Discounts resent input and lets the provider skip most of the prefill work | One prefix edit, one reordered schema, or a compaction turns the win off silently — monitor hit rate as a metric |
Trim tool output at the source | Both | Smaller transcript means fewer billed tokens on every later call and a faster prefill | Trimming the field the model needed. Trim in the tool with a documented shape, not by truncating text mid-record |
Shorter model turns (output caps, “be terse”) | Both | Decode is sequential and dominates per-turn wall clock; verbose turns are billed as output, then again as input forever after | Truncated work if the cap is too tight — cap generously and instruct for brevity, rather than the reverse |
Parallel tool calls | Latency | Independent calls in one turn execute concurrently: their time becomes the max instead of the sum, and several round-trips collapse into one | Concurrent writes and ordering bugs. Parallelise reads freely; serialise anything with side effects |
Model routing | Both | Cheaper models are usually also faster per token | Quality dip on misrouted turns; escalations billed twice; splits the cache |
Compaction | Both, in long runs only | Shrinks the payload every later call resends and prefills | Lossy summaries; an extra call; cache invalidated from the rewrite point |
Speculative / prefetched work | Latency — at a cost premium | Start the likely next tool call, or run cheap and strong models concurrently, before you know which you need | Pure waste on every miss (see the warning below), and never safe for side-effecting actions |
Batch endpoint | Cost only — spends latency | You trade scheduling flexibility for a discount | Unusable for an interactive turn; right for evals, judging, and backfills |
Streaming | Neither — perceived only | Shows tokens as they are produced | Nothing directly — but it flatters a slow agent and can stop you fixing the real TTFD problem |
Key terms: time to first token, time to done, prefill, streaming, parallel tool calls, speculative execution
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.