Harness Mechanics
Lesson 3 of 4 in Building an Eval Harness.
An eval run is an experiment, and experiments earn trust by being repeatable. The harness’s first job is to pin every variable that isn’t the one you changed: the model version (a pinned snapshot, never a floating alias), the prompt template, the Sampling settings, the golden-set version, and the grader — code, judge model, and rubric alike.
One pin deserves special suspicion: Temperature. Setting it to 0 removes sampling randomness but is not a hard determinism guarantee — serving stacks introduce their own nondeterminism through batching and floating-point ordering, and some models are designed to sample. Where the API supports a seed parameter, set it; treat it as reducing nondeterminism, not eliminating it. The mechanics of why live in Decoding and Sampling.
The practical consequence: when variance matters, run the set N times — both variants, same items — and look at the spread before reading any single number.
| What to pin | Why | What breaks if you don’t |
|---|---|---|
Model version | Providers update models behind floating aliases; the model is the thing under test | Scores Drift with no change on your side — you debug a “regression” that is a silent model swap |
Prompt template | The prompt is part of the system under test; a one-line tweak is a new experiment | You can’t tell whether a delta came from the model or a teammate’s wording edit |
Sampling settings | Temperature and top-p reshape the output distribution; scores only compare at fixed settings | Sampling noise masquerades as quality change, in either direction |
Golden-set version | Adding or relabeling items changes what the number means | 82% on v3 vs 79% on v2 compares two different exams and calls it progress |
Grader version | Grader code, judge model, and rubric are measurement instruments | A stricter judge model reads as a quality regression in the system under test |
Seeds & run count | Seeds (where supported) reduce nondeterminism; a fixed N of repeated runs reveals the spread | A single run’s wobble gets read as signal — the most common eval mistake there is |
The second job: log per-item results, not just aggregates. For every item, every run, record the item ID, the exact input, the raw output, the verdict, and — for judge rungs — the judge’s justification. Aggregates feed dashboards; per-item records feed diagnosis. Without them you cannot answer the only questions that matter after a score moves: which items flipped, and why. Lesson four runs on this data.
The third job: cost the harness before you schedule it. The arithmetic is items × runs × (tokens per item, input and output) — doubled when a judge reads every output. A 200-item set, run 5 times with judge grading, is a thousand system calls plus a thousand judge calls, every time someone edits a prompt. That is usually still cheap next to one production incident, but know the number: it decides whether the full set runs on every commit or nightly, with a small smoke slice on each change.
Tool: Token Cost Estimator — Put numbers on your own harness: the Token Cost Estimator turns items × runs × tokens (plus judge overhead) into a per-eval-run figure you can budget around.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.