Pass rates and honest variance
Lesson 5 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.
You have cases and you have graders. The output is a number — and the number is where people fool themselves.
Run each case K times, not once. A case that passes 3 times out of 5 is not a passing case and not a failing case; it is a 60% case, and that fact is the most useful thing you know about it. K = 3 is the practical floor for a development loop, K = 5 to 10 for anything gating a release. Report per-case rates and the suite rate, because they answer different questions: the suite rate tells you whether to ship, the per-case rates tell you what to fix.
Then be honest about noise. If your suite is 20 cases run once each, a two-case swing is a 10-point move in your headline number and means essentially nothing. Before you celebrate a 4-point improvement, you need to know how much your suite moves when you change nothing at all — run the baseline twice and look. That is your noise floor, and improvements smaller than it are stories, not results.
| What is being claimed | The claim as usually made | What is missing | The honest version |
|---|---|---|---|
Quality | “The agent is 94% accurate.” | Dataset, case count, runs per case, grader, which slices. | “86% suite pass rate on 50 real cases × 5 runs, deterministic outcome checks plus hard trajectory assertions; 94% on the happy-path slice, 61% on the multi-intent slice.” |
An improvement | “The new prompt is better.” | Baseline, delta, noise floor, and what got worse. | “+7 points on the escalation slice (72% → 79%), noise floor ±3 on repeat runs; refund slice unchanged; step count up 15%.” |
A failing case | “Case 17 is flaky.” | The rate, and whether the flakiness is the agent or the grader. | “Case 17 passes 2/10. The two passes take a different tool path — this is agent variance, not grader noise, and the failing path is the one to fix.” |
A model upgrade | “New model, same behaviour.” | A re-run of the whole suite, including the judge’s calibration. | “Suite 86% → 88%; two safety cases regressed from 10/10 to 7/10; judge agreement with human labels dropped 4 points, so the judge rubric needs re-calibration before these numbers are comparable.” |
Readiness | “It is ready for production.” | The autonomy level the number licenses. | “86% is ready for suggest-only. The gate stays until the safety slice is 100% over 10 runs and we have a month of shadow-mode data.” |
Aggregate rates hide the failures you care about most
A suite at 92% overall can be a suite that fails every single safety case, if safety cases are 6% of the dataset. Averages are where dangerous failures go to hide.
Report by slice — intent, customer tier, language, tool path, and always a separate safety/adversarial slice — and set different thresholds per slice. The safety slice is not allowed to trade points for helpfulness anywhere else.
Distinguish agent variance from grader variance
When a case wobbles, ask which component is wobbling. Re-grade the same stored run several times: if the verdict changes, your grader is the noisy part — usually a judge with a vague rubric. If the verdict is stable but the runs differ, the agent is the noisy part.
The two have completely different fixes: tighten the rubric versus fix the plan, the tool description, or the context. Teams that skip this diagnosis spend weeks tuning an agent to satisfy a coin-flip grader.
Do not compare numbers across changed conditions
A pass rate is only comparable against another pass rate produced with the same dataset version, the same grader version, the same model, and the same tool set. Change any of them and you have a new metric wearing the old metric’s name.
Version everything and record it alongside the result: dataset hash, prompt version, judge rubric version, model id. This is the same provenance discipline the deployment modules apply to releases, and it is what makes a chart of pass rate over time mean anything at all.
Cost and latency are eval results too
A prompt change that adds 7 points of pass rate and doubles token spend is a trade-off, not a win — and one you can only see if the suite records cost and latency per run alongside the verdict. Capture tokens, wall-clock time and step count in the same report as the rate; the cost-and-latency module turns those into budgets.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.