Why “it worked when I tried it” is not evidence
Lesson 1 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.
Someone on your team just demoed an agent. It handled the refund case beautifully. The room nodded. Ship it?
You have exactly one data point, drawn from a distribution you have not measured. Non-determinism means the model samples its next token from a probability distribution — so it samples its next action from one too. Run the same task again and the trace may branch differently at step three: a different tool, a different query, a different answer. Your demo was not a test. It was one draw.
This is the whole reason the evals domain exists. In deterministic software, one passing run of a unit test is a proof about a code path — the same input takes the same path forever. In agent software, one passing run is an anecdote about a distribution. The unit of evidence changes from pass/fail to pass rate, and everything downstream — how you debug, how you review a pull request, when you dare raise the autonomy dial — changes with it.
| Dimension | Unit test (deterministic code) | Eval (agent) |
|---|---|---|
What one run tells you | Everything. Same input → same path → same output, forever. | Almost nothing. One sample from a distribution that may contain failures you did not draw. |
Verdict shape | Boolean: pass or fail. | A rate with variance: 43/50 passed, 86%, ±5 points run to run. |
What you assert on | Exact return value; exception type. | The final answer and the path taken — see the outcome/trajectory split in lesson three. |
Why it fails | A logic bug. Deterministic, reproducible, bisectable. | A bad tool description, a poisoned retrieval, a wandering plan — often unreproducible, which is why you keep the traces. |
What “fixed” means | The red test is green. Done. | The pass rate moved up on the cases you targeted and did not drop anywhere else. Regression pressure is constant. |
Cost per run | Microseconds, free. | Seconds to minutes, and real money in tokens — which is why dataset size is an engineering decision, not a maximisation problem. |
Key terms: eval, golden dataset, non-determinism, pass rate, trace, LLM-as-judge
Eval-first development inverts the usual order. The instinct is to build the agent, then — once it feels good — bolt on measurement. Teams that do this discover they cannot tell whether their prompt change helped, because they have nothing to compare against. They tune by anecdote for months.
Eval-first means the dataset comes early and stays in front of the work: capture the cases, define what “good” means for each, get a baseline number even if it is embarrassing (30% is a fine starting pass rate — it is a number), then change one thing at a time and watch the number. This is not process theatre. It is the only mechanism that tells you whether your last four hours of prompt editing were an improvement or a lateral move.
The eval-first loop
- Collect real cases
From production traffic, from support tickets, from every failure anyone has seen. Twenty is enough to start.
- Define expected outcome per case
The ground truth: the right answer, or the properties a right answer must have, plus any action the agent must never take on this case.
- Run the suite → baseline rate
Your first number. Low is fine; unknown is not. Record which model, which prompt version, which tool set produced it.
- Change exactly one thing
One prompt edit, one tool description, one model swap. Two changes at once and you learn nothing about either.
- Re-run · compare rates
Better, worse, or inside the noise? The variance of the suite decides whether a 3-point move means anything at all.
- Keep it · new baseline
- Revert · log what you learned
A failed experiment with a recorded number is still progress. An unmeasured change that “felt better” is technical debt with confidence.
- Suite runs in CI on every change
The endpoint of the discipline: the suite is no longer something you run, it is something that runs on you.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.