The bake-off: cost, latency, and quality on one table
Lesson 5 of 5 in Cost and Latency Budgets You Can Defend.
Every lever in this module trades one axis for another, which means optimising any single axis blind is how you ship a cheap agent that does not work. The discipline that prevents it is the bake-off: run each candidate configuration over the same golden set and record cost, latency, and quality together, per run.
You already have the quality machinery — that is the eval harness from Eval Fundamentals, and the per-run cost and timing numbers come straight out of the trace you already collect. A bake-off is mostly wiring you own, plus one new metric worth naming:
cost per successful run = cost per run ÷ pass rate
That metric exists because a failed run is not free. Somebody retries it, or a human does the task by hand, or a customer leaves. A configuration that is 40% cheaper and 20% worse usually costs more per unit of work delivered — and only this metric shows it.
| Configuration | Pass rate (n=120) | p50 time-to-done | p95 time-to-done | Cost / run | Cost / successful run |
|---|---|---|---|---|---|
A — strong model everywhere, no caching | 0.91 | 105 s | 240 s | $0.42 | $0.46 |
B — A plus prompt caching | 0.91 | 88 s | 205 s | $0.15 | $0.16 |
C — B plus routing (cheap default, escalate on signals) | 0.87 | 71 s | 260 s | $0.09 | $0.10 |
D — cheap model everywhere, caching, compaction at turn 6 | 0.64 | 55 s | 120 s | $0.05 | $0.08 |
Read that table in a fixed order and it makes the decision for you.
Apply the quality floor first — it is a constraint, not an axis. D is out at 0.64, despite the best cost per successful run on the table. If you let cost-per-success rank configurations on its own, it will happily recommend an agent that fails a third of the time, because the metric cannot know that a 36% failure rate is a product failure. Floor first, then optimise.
Take the free wins. B beats A on every column at once — same pass rate, faster, a third of the cost. That is what a dominating configuration looks like, and it is why prompt caching goes in before you argue about models. Interesting trade-offs only start after you have banked the dominating ones.
Then decide against your binding constraint. C looks tempting: cheaper, and 17 s faster at p50. But its p95 is 260 s, which breaks the 3-minute SLO, because escalations add a whole extra turn to the hardest tasks — the exact tasks already sitting in the tail. And its 4-point pass-rate gap on 120 tasks is about five tasks: too close to call without repeat runs. So ship B, keep C as a candidate, and go widen the golden set. The table did not just pick a config; it told you which measurement to improve next.
“We compared them on five tasks”
Five tasks cannot distinguish a 4-point pass-rate difference from noise — and neither can 120, really, which is why the honest conclusion above was “too close to call, widen the set.” Cost and latency stabilise on far fewer runs than quality does, so a small bake-off can legitimately tell you which config is cheaper while telling you nothing about which is better. Report the sample size next to every pass rate, and repeat runs on the same tasks to see the run-to-run variance before you attribute a gap to your change.
“Caching made it slower in our test”
You probably measured the cold run. The first execution of a configuration pays cache writes — sometimes at a premium — and gets no hits. Steady-state behaviour needs a warm-up pass, and cache time-to-live means a low-traffic agent may be cold on most real runs anyway. Measure and report both numbers: cold-start and warm. Which one matters is a traffic question, not a benchmarking preference.
“Median latency improved, so we shipped it”
Agent latency distributions have fat tails, because the number of turns varies per task. The p50 is set by the easy tasks; the p95 is set by the ones that loop, retry, escalate, or hit a slow tool — and it is the p95 that trips your timeouts, breaks your SLO, and generates the support tickets. Configuration C above is faster at p50 and unshippable at p95. Always quote both.
“We priced it from the vendor’s rate card”
Multiply your estimated tokens by list prices and you will be wrong in both directions: cached and uncached input are separate line items at different rates, retries and failed calls still bill, and your fan-out is probably larger than you think. Read the usage fields of every response and total them per run id. The gap between a rate-card estimate and measured usage is routinely 2× or more, and the direction of the error is not predictable.
“We forgot to count the judge”
An LLM judge scoring 120 traces is itself a token bill, and a trajectory judge that reads whole traces can cost more per task than the agent did. That is often worth paying — but budget it explicitly, sample production traffic rather than scoring all of it, and consider cheap deterministic checks for the criteria that do not need a model at all.
“We changed the model and the prompt in the same config”
Then you learned that the pair differs from the baseline, and nothing about which half did the work — or whether they cancelled out. One variable per row. It is slower and it is the only version of this exercise that produces a transferable conclusion.
Tool: Eval Suite Builder — Build the bake-off: assemble a golden set, attach cost and latency meters, and see which configuration survives the quality floor.
Interactive flashcard deck.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.