The cost levers, honestly sized
Lesson 2 of 5 in Cost and Latency Budgets You Can Defend.
There are only a handful of real levers, and each one has a price paid in a currency other than money: engineering time, quality, latency, or detail. What follows sizes each lever against the same 10-turn baseline from lesson 1 — 120,000 input tokens, 4,000 output, about $0.42 per run at illustrative prices — so you can compare them honestly instead of collecting optimisations.
Start with the two levers that cut tokens before you get clever: trim what tools return, and delete what the prefix carries. They are boring, they need no vendor features, and they usually improve quality. Everything below is what you reach for after those.
Prompt caching
What it is. Providers will store the processed form of a prompt prefix and charge a steeply discounted rate to re-read it on a later call — often on the order of a tenth of the normal input price (illustrative; discounts, minimum cacheable sizes, and time-to-live differ by vendor and change, so check your provider’s current docs).
Why agents are the ideal customer. An agent’s transcript is append-only: call n’s payload is call n−1’s payload plus a new turn. On the worked run, 99,000 of the 120,000 input tokens are re-reads of bytes already sent — around $0.15 instead of $0.42 per run.
What it does NOT do.
- It does not give the model memory. The model is still stateless; caching discounts the resend, it does not remove it.
- It only matches a prefix, and usually only a byte-identical one. Insert a timestamp, reorder tool schemas, or edit one system-prompt word and everything after the change is a miss.
- It does not shrink your context window usage. Cached tokens still occupy the window.
- Writes are not always free — some providers charge a premium to populate the cache, so a run that never gets a second call can cost more.
Design rule. Order the context by volatility: stable system prompt → tool schemas → retrieved reference material → conversation → the volatile bit (timestamps, per-turn instructions) last.
Model routing
What it is. Do not send every turn to your best model. Use a cheaper model as the default and escalate on a signal: the task class up front, or mid-run evidence — low self-reported confidence, a tool call that failed twice, a diff touching sensitive code, a structured output that failed schema validation.
Honest sizing. Routing 7 of 10 turns to a model at a fifth of the price took the worked run from $0.42 to about $0.21. But savings track the share of tokens on cheap turns, not the share of turns — a cheap model still resends the whole fat transcript, so routing the late, large turns saves far more than routing the early, small ones.
What it costs you.
- Escalations pay twice. A turn attempted cheap and redone strong costs both. If escalation fires on 30% of turns your savings shrink faster than the arithmetic suggests.
- It splits your cache. Two models mean two caches; alternating between them mid-run can throw away most of your caching win. Routing and caching genuinely fight each other — measure the combination, not each lever alone.
- Quality is not uniform. Cheap models often hold up on summarising and extraction and fall over on multi-step planning and tool selection. Route by task shape, and prove it on your golden set.
- A router that is itself an LLM call adds a turn. Prefer cheap deterministic signals where they work.
Compaction
What it is. When the transcript crosses a threshold, replace the old middle of it with a summary — findings so far, decisions made, open questions, files touched — and keep going from the smaller context. This is the token-budget face of the compaction and context-engineering patterns the architectures domain covers in depth.
Honest sizing. Compacting once at turn 6 of the worked run (13,000 tokens down to a 5,000-token prefix-plus-summary) saves about 40,000 input tokens but adds a summariser call, netting roughly $0.35 instead of $0.42 — about 17%. Underwhelming at 10 turns.
At 40 turns the same policy applied every 10 turns is the difference between ~1.68M input tokens and roughly a third of that, and it is what keeps the run inside the window at all. Compaction is a long-run lever; do not bother reaching for it on short ones.
What it costs you.
- Lossy by construction. The summariser decides what mattered, and it will sometimes drop the detail the agent needed on turn 30. Keep raw artifacts (file paths, ids, exact error strings) verbatim and summarise the prose around them.
- It busts the cache from the rewrite point onward — you just changed the prefix.
- The summariser is itself a model call that can be wrong, and its errors are invisible afterwards because the evidence is gone. Log both sides of every compaction in the trace.
Batching
What it is. Submit work to an asynchronous/deferred endpoint and collect results later. Providers commonly discount this substantially — figures around half price are typical (illustrative; verify the current terms and the service window your provider commits to) — because you are letting them schedule your work.
Where it applies. Almost never to a live agent turn: the loop cannot proceed until the call returns, and batch turnaround is measured in minutes to hours. Batching pays for the work around the agent:
- Eval runs. Scoring 300 golden tasks nightly is the ideal batch workload — and evaluation is often where a serious team’s token spend actually lives.
- LLM-as-judge scoring, backfills, re-labelling, offline classification, and nightly triage where the queue has hours of slack.
What it costs you. Latency, and therefore interactivity. Also complexity: you now need job tracking, partial-failure handling, and a plan for results that arrive after the situation changed. Batch also does not remove the resend tax — it just discounts it.
Related lever. Some platforms let you run evaluation in batch mode against stored traces rather than re-running the agent; AWS documents on-demand, online-sampled, and batch modes for Bedrock AgentCore Evaluations, for example. Verify the current capabilities of whatever platform you use.
| Applied to the baseline run | Billed input | Run cost | Change | Paid for in |
|---|---|---|---|---|
Baseline — no levers | 120,000 at full rate | $0.42 | — | — |
Prompt caching (stable prefix, append-only transcript) | 21,000 full + 99,000 cache reads | ~$0.15 | −64% | Cache misses on any prefix edit; possible write premiums; TTL expiry between runs |
Trim tool output (avg result 1,600 → 700 tokens) | 79,500 at full rate | ~$0.30 | −29% | Per-tool engineering; the risk of trimming the field the model needed |
Compaction once at turn 6 | 80,000 + ~13,000 for the summariser | ~$0.35 | −17% | Lost detail, one extra call, cache invalidated from the rewrite point |
Model routing — 7 cheap turns, 3 strong | 120,000, most at a fifth of the rate | ~$0.21 | −50% | Quality dip on misrouted turns; escalations billed twice; the cache splits in two |
Batch endpoint (non-interactive work only) | 120,000 at the batch rate | ~$0.21 | −50% | Minutes-to-hours turnaround — unusable for a live turn |
Turn cap 10 → 6 (the blunt lever) | 48,000 at full rate | ~$0.18 | −57% | Unfinished runs. Only defensible once you have measured how many tasks actually need turns 7–10 |
Key terms: prompt caching, model routing, compaction, batch inference, context engineering, stopping condition
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.