Benchmarks, their limits, and evaluating generative systems
Lesson 4 of 6 in Testing and Evaluation: Fairness Math, Red-Teaming, and Explainability.
A benchmark is a measurement instrument, and instruments decay. Governance teams that read vendor eval tables need to know the three ways a benchmark stops measuring what it claims — because “state of the art on the leaderboard” is routinely offered as assurance evidence, and routinely means less than it seems.
Contamination — the test questions were in the training data
Frontier models train on internet-scale corpora that contain the benchmarks themselves — questions, answers, and discussion threads about them. A model can “ace” a bar-exam benchmark by recall, not reasoning. Detection is genuinely hard (paraphrased leakage evades string matching), which is why serious evaluators maintain held-out, private test sets and rotate them, and why regulators increasingly ask how an eval score was produced, not just what it was.
Goodhart’s law — when a measure becomes a target, it ceases to be a good measure
Labs optimise toward the public benchmarks that drive rankings and procurement. Capability then concentrates exactly on the benchmark distribution and thins out beside it. The governance translation: a benchmark score predicts performance on the benchmark, and its transfer to your use case is a hypothesis you must test, not an inference you may draw.
Saturation — the ceiling makes differences meaningless
Once top models cluster above ~90% on a benchmark (as happened to MMLU and its generation of tests), remaining differences are noise, label errors, and prompt sensitivity. Saturated benchmarks keep circulating in marketing long after they stop discriminating between systems — check the date and the ceiling before treating a score as information.
The deeper limit — a leaderboard is not a fitness-for-purpose test
Even an honest, unsaturated benchmark measures average performance on generic tasks. Your deployment has a specific population, domain vocabulary, failure-cost structure, and adversary. The only evaluation that supports a deployment decision is a task-specific eval on your own data with your own error costs — the benchmark just tells you which models are worth that effort.
Generative systems break classical evaluation — there is no single right answer to score against. The field’s response, visible in Stanford’s HELM project, is multi-metric evaluation: accuracy alongside robustness, calibration, toxicity, bias, and efficiency, reported per scenario rather than as one number. For deployments, that generic grid gets replaced by task-specific eval suites: curated golden sets, adversarial cases from red-teaming, and regression cases from production incidents.
Scoring open-ended outputs at scale usually means LLM-as-judge — one model grading another’s answers. It is cheap and correlates tolerably with human raters on many tasks, and it fails in patterned ways a governance reviewer must know: self-preference (judges favour outputs resembling their own style, including their own model family), position bias (favouring the first answer shown in pairwise comparisons), verbosity bias (longer answers score higher at equal quality), and shared blind spots (a judge from the same training distribution misses the same subtle errors). Controls: randomise positions, use rubric-anchored scoring rather than free preference, calibrate the judge against a human-rated sample, and never let the judge model family evaluate itself for a release decision.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.