The golden dataset: twenty real cases beat a thousand invented ones
Lesson 2 of 6 in Eval Fundamentals: You Cannot Improve What You Cannot Measure.
A golden dataset is a set of cases with verified expected results. For an agent, each case is usually four things: the input (a ticket, a request, a repo state), the starting context, the expected outcome, and the actions that would be unacceptable regardless of outcome.
The first question everyone asks is “how many?” The answer is uncomfortable: fewer than you think, and they must be real. Twenty cases harvested from actual traffic and actual failures will teach you more than a thousand you invented at your desk, because the ones you invent come from your mental model of the system — and your mental model is precisely what is wrong. The distribution of things users actually do is stranger than the distribution you can imagine: the mis-spelled order number, the two questions in one message, the customer replying to a three-month-old thread.
Source 1 · Production traffic — the boring middle
Sample real runs and keep the ones that are representative, not the ones that are interesting. You need the mundane majority in the set, otherwise your pass rate is computed entirely on hard cases and every improvement looks like noise.
Practical method: stratified sampling from your trace store. Bucket runs by intent, tool path, and stop reason, then take a few from each bucket. Redact before storing — an eval dataset is a permanent copy of customer data unless you make it not one.
Source 2 · Every incident, forever — the highest-value cases you will ever get
Something went wrong in production. Somebody wrote a post-mortem. The case goes in the dataset before the fix ships. This is the single habit that separates teams whose agents get steadily better from teams who keep re-fixing the same behaviour.
The rule has a name in ordinary software — a regression test — and the discipline transfers exactly. The difference is that your case may only fail 30% of the time, so it needs multiple runs to be a reliable sentinel.
Source 3 · Near-misses and human overrides
Every time a reviewer rejects a proposed action at an approval gate, that is a labelled failure with free ground truth attached: a human already decided what should have happened. Same for edits — the reviewer who fixes the draft before sending is annotating your dataset for you.
Mine the override log. It is the cheapest labelled data an agent team owns, and almost nobody harvests it.
Source 4 · Adversarial and safety cases
Cases that probe the boundary: instructions hidden in a retrieved document (prompt injection), a tool that returns nonsense, a request the agent is not authorised to fulfil. The expected result here is usually a refusal, an escalation, or a stopped run — not an answer.
These belong in the same suite as your happy-path cases so that a “helpfulness” prompt tweak cannot quietly trade safety for pass rate without showing up as a red row. The security domain builds these out properly in guardrails and injection work; the eval suite is where they get enforced on every change.
Source 5 · Synthetic cases — useful, and a trap
Generated cases are legitimate for coverage of shape: fifty variations of date formats, twenty phrasings of the same intent, a long-context stress case. They are cheap and they scale.
The trap is grading yourself on a world you invented. If synthetic cases dominate your suite, your pass rate measures agreement with your own imagination. Keep them clearly labelled as a separate slice, and never let them outvote the real ones in your headline number.
Ground truth is the part that rots. Someone writes “expected: full refund” in January; in March the refund policy changes and nobody updates the case. Now the suite is failing runs that are correct and passing runs that are wrong — and worse, the team learns to ignore red rows. An eval suite people have learned to ignore is more dangerous than no suite, because it still looks like diligence.
Treat ground truth as owned data: every case carries who verified it, when, and against which policy version. When policy changes, the cases touching that policy get re-verified as part of the change. Cases whose expected result is genuinely ambiguous do not get a coin-flip answer — they get a property instead ("must cite a policy section", "must not promise a date"), which is a real assertion rather than a guess dressed as one.
Interactive sorting exercise: You are assembling the first golden dataset for a customer-support agent. Sort each candidate case: does it belong in the suite as-is, does it need work before it can be a case, or does it not belong at all?
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.