Latency and cost: the two meters that gate every design
Lesson 5 of 5 in LLM Mechanics for Agent Builders — No Math Required.
Two meters run during every agent task: dollars and seconds. You have met the dollar meter. The seconds meter has its own anatomy. A call’s latency is time to first token — which grows with how much input the model must ingest — plus generation time: output arrives sequentially at some tokens-per-second rate (tens to low hundreds per second is a reasonable 2026 intuition; illustrative, model- and load-dependent). Reading is fast and parallel; writing is the slow part of a single call.
Now multiply by the loop. An agent run is sequential: call, tool, call, tool — no step can start before the previous one returns. Twenty calls at 2–8 seconds each, plus tool execution (test suites, API calls, browsers), lands a run in the minutes. That single fact decides where agents are viable: nobody waits three minutes for autocomplete, but three minutes is a bargain for a fixed bug or a researched brief.
| Meter | Per call | Per run of N turns |
|---|---|---|
Cost | Input tokens × input price + output tokens × output price (output per-token price typically 3–5× input). | Sum over N calls of a growing input — roughly quadratic in N. Doubling turns ≈ 4× cost, not 2×. |
Latency | Time to first token (grows with input length) + output tokens ÷ generation speed. | Sequential sum of every call plus every tool execution. The slowest tool is often the real bottleneck. |
What blows it up | Bloated system prompt and tool schemas; a model reasoning at length before answering. | Unbounded loops: an agent retrying its way through failures, or wandering without a tight stopping condition. |
First lever to pull | Shrink the always-resent prefix; trim tool outputs before appending them. | Cap turns and tighten the stopping condition — then route cheap steps to a smaller model. |
Your agent’s cost per task tripled — where do you look first?
Interactive decision tree — outcomes:
- Tighten the loop’s exits
Cap turns, fail fast on repeated tool errors, and make the stopping condition concrete (“tests green” beats “task seems done”). Flailing runs burn quadratic money to accomplish nothing — cutting them is pure win on cost, latency, and reliability.
- Split the task, not the budget
If the work genuinely grew, decompose it: shorter runs with fresh contexts beat one marathon transcript — the resend bill resets, and reliability decays less over fewer steps. The architectures domain covers the patterns.
- Shrink and cache the prefix
Every schema and system-prompt paragraph is billed on every call of every run. Deregister tools the agent doesn’t use, tighten the prompt, and turn on prompt caching so the unchanged prefix is re-read at the discounted rate.
- Trim tool output at the source
Return the 20 matching lines, not the whole file; summarise the API response; paginate. A tool result is paid for once when it arrives and then again on every later call — the cheapest token is the one that never enters the transcript.
Key terms: latency, time to first token, prompt caching, stopping condition, token
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.