Your Golden Set

Lesson 1 of 4 in Building an Eval Harness.

Public benchmarks measure the field; they do not measure your task. A strong MMLU score says nothing about whether a model summarizes your support tickets, extracts your invoice fields, or refuses your out-of-policy requests. The previous modules ended at the same conclusion from different directions: leaderboards saturate, test sets leak, judges have biases — so the number you can actually trust is one you built yourself.

That number rests on a Golden set: a frozen, labeled sample of your real work, with a known right answer (or a written definition of “good”) for every item. Around it you build an Eval harness — the runner that feeds each item to the system, collects outputs, applies a Grader, and reports results. This lesson builds the set; the rest of the module builds the machinery.

Where do the items come from? Real traffic, sampled honestly. The temptation is to collect the cases you know the system handles — demos, happy paths, the examples in your prompt. Resist it. Sample across the real distribution: every major intent, short inputs and long ones, every language you serve, clean text and the messy paste-dumps users actually send. Then deliberately over-sample the tail: past failures from your bug tracker, edge cases you fear, inputs that once triggered a Hallucination. Regressions live in the tail; a set without a tail cannot see them.

From traffic to a versioned golden set

  1. Real production traffic

    Logged inputs (with consent and PII handling), plus known failures from bug reports and support escalations.

  2. Sample across the distribution

    Stratify by intent, length, language, and difficulty. Over-sample known-hard cases — the tail is where regressions hide.

  3. Label with written criteria

    A short rubric per item type: what counts as correct, what counts as failure. Two labelers should reach the same verdict from the text alone.

  4. Freeze the set

    Items and labels stop changing silently. Any fix or addition is a deliberate, recorded change.

  5. Hold a slice back

    A reserve you rarely run, so prompt iteration can’t quietly overfit to the whole set.

  6. Versioned golden set (v1, v2, …)

    Every reported score names its set version. New traffic patterns feed the next version.

Label with written criteria, not vibes. For each item type, write down what a pass requires — “the summary names the customer’s actual problem and the resolution; invented details are an automatic fail.” The test of a good criterion is inter-annotator agreement: two people labeling independently should reach the same verdict. If they cannot, your eval will inherit that ambiguity as noise, and later, any LLM-as-judge you build will inherit it as bias. Written criteria are also reusable capital — in the next lesson they become your judge Rubric almost verbatim.

Size the set honestly — in orders of magnitude, not false precision. Dozens of well-chosen items will catch obvious regressions: a prompt edit that breaks JSON output, a model swap that changes refusal behavior. Comparing two closely matched models, or detecting small quality shifts, takes hundreds — small sets make small deltas unreadable, a point lesson four makes concrete. Start small and honest, grow with your traffic, and never let the set’s size imply a precision it does not have.

Version it like code, and hold some back. The set is frozen per version: scores are only comparable when the exam is identical. And keep a held-out slice you touch rarely. The reason is Contamination at personal scale: iterate a prompt against the full set long enough and you are fitting to those fifty items, not to the task — the same overfitting-to-the-test-set failure that haunts public benchmarks, reproduced on your laptop.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.