Interpreting Results
Lesson 4 of 4 in Building an Eval Harness.
The harness has run. A number came out. The last skill is refusing to be fooled by it.
Start with the distinction the whole lesson turns on: aggregate deltas versus per-item flips. The aggregate — 84% became 86% — is a summary that compresses away the very information a decision needs. The per-item view — which items changed verdict, in which direction — is where the truth lives. Two situations make the difference vivid. An aggregate that moved may be nothing but wobble: on a 50-item set, 2 points is one item, and repeated runs of an unchanged system routinely differ by that much. And an aggregate that didn’t move may hide real change: six items newly passing and six newly failing net to zero, yet if the new failures cluster in your highest-stakes intent, the “neutral” change is a regression.
So the reading order is fixed: first, is the delta bigger than the noise band you measured by rerunning? Second, which items flipped, and do the flips cluster? Only then does the aggregate mean anything.
Line chart with two series, Prompt A and Prompt B, showing accuracy percent on the y-axis from about 72 to 82, across five identical reruns on the x-axis. Prompt A scores 76, 80, 74, 78, and 76; Prompt B scores 78, 74, 80, 76, and 78. The two lines cross repeatedly and occupy the same range, illustrating overlapping noise bands.
Error analysis beats score staring. The score tells you that something is wrong; only the failures tell you what. The loop is unglamorous and it is where all the improvement comes from: pull the per-item log, read every failure — the actual input, the actual output, the grader’s verdict and justification — and tag each with a failure mode in plain words: “invented a policy number,” “ignored the second question,” “broke JSON on long inputs,” “refused a benign request.” Tags accumulate into clusters; the biggest cluster is your next fix. One targeted change against a named failure mode routinely does more than ten rounds of untargeted prompt fiddling — and after the fix, the same harness verifies it and catches what it broke.
Two habits keep the loop honest. Read failures before forming a theory — the log regularly refutes the explanation you were sure of. And read a few passes too, especially near-misses your Grader barely accepted: grader bugs hide among the passes, where nobody looks.
Variance intuition, without fake statistics
You do not need formulas to reason soundly about eval noise — you need three habits.
Small sets wobble; know your granularity. Each golden-set item contributes one all-or-nothing verdict, so a 50-item set moves in 2-point steps: a 2-point delta is one item. One borderline output, one grader hiccup, one sampling accident — any of them produces the entire “improvement.” The same delta on a 1,000-item set is twenty items, a far harder thing for chance to arrange.
Measure your noise band instead of assuming one. Rerun the unchanged system several times with every pin fixed. The spread you observe — however wide it is — is your noise band, measured under your model, your grader, your serving stack. Deltas inside the band are “no signal yet”; deltas well outside it deserve attention. This one habit replaces most of the statistics you are tempted to skip.
Compare paired, per item. Two variants run on the same items can be compared flip by flip, which is far more sensitive than comparing two aggregates: if B fixes nine items A failed while breaking two, that pattern is informative even when the aggregates barely differ — and the two breakages are your next bug reports.
Formal machinery for all of this exists — paired tests, bootstrap resampling over items — and a statistician can apply it properly to eval data. The trap is the middle path: plugging one noisy run per variant into a textbook formula whose assumptions (independent items, deterministic measurement) your harness visibly violates, then shipping the resulting p-value as truth. When a decision matters and the noise band swallows your delta, the honest moves are to grow the set, raise the run count, or accept that the difference is too small to act on.
In production
The harness is not a one-off study — it is the asset every later change gets gated on. The Adapting LLMs domain assumed you had one: prompt edits, model swaps, RAG changes, and fine-tunes all rerun it before they ship, and the Production Evals module turns that rerun into a formal gate. The clouds ship managed versions of the same loop — dataset in, graders over outputs, per-item results out — so the discipline transfers, not just the tooling.
AWS
Amazon Bedrock offers model-evaluation jobs: you supply a prompt dataset (your golden set), choose automatic metrics, model-as-judge graders, or human-review workflows, and the service runs the candidate model over the items and reports results. The mechanism is exactly this module’s — dataset, grader, runner, report — so golden-set versioning and held-out discipline carry over unchanged.
Azure
Azure AI Foundry’s evaluation tooling runs built-in and custom evaluators — programmatic checks and rubric-guided judge evaluators — against your dataset, logging per-item rows so prompt and deployment variants can be compared side by side. Wired into CI, it becomes the regression gate this module builds toward: no change promotes without a passing run.
Google Cloud
Vertex AI’s generative-AI evaluation service supports computation-based metrics and model-based grading, pointwise or pairwise, against your own dataset. Pairwise mode is the managed form of this lesson’s paired comparison: same items, two candidates, a verdict per item — flip analysis as a service rather than aggregate staring.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.