What a Benchmark Is
Lesson 1 of 4 in Benchmarks and Their Limits.
A Benchmark looks like a number — “the model scored 87” — but it is actually a machine with four parts, and every part is a design choice someone made.
The task set is a fixed collection of inputs: exam questions, programming problems, word problems. Ideally it is a Held-out set — data the model never trained on, so performance reflects ability rather than memory. (Whether that ideal survives web-scale training is the next module’s subject: Contamination.)
The answer key defines what counts as correct: a labeled option, a reference number, a suite of unit tests. Answer keys are built by humans at scale, which means they contain some errors and ambiguities — a fact that will matter enormously when we reach saturation.
The metric turns raw model outputs into the score: accuracy over multiple-choice picks, exact match on a final answer, pass rates over executed code. The metric decides what the benchmark can and cannot see — a metric that only checks the final number is blind to how the model got there.
The harness is the software that actually runs the model over the tasks: the prompt template, how many worked examples (“shots”) are shown, Sampling settings like Temperature, and how an answer gets extracted from free text. This is the part most people forget exists — and it moves scores all by itself. The same Eval harness idea scales down to your own product evals later in this domain.
Four names come up in nearly every model announcement, and each probes something different.
MMLU (Hendrycks et al., 2020) tests broad knowledge: multiple-choice questions across 57 subjects, from law and medicine to mathematics and US history. The model picks one of four options; the metric is accuracy. It was designed to be hard for models of its era across breadth no single fine-tune could cover.
HumanEval (Chen et al., 2021) tests code generation: 164 hand-written Python problems, each a function signature and docstring plus hidden unit tests. Generated code is executed — the metric, pass@k, is the fraction of problems for which at least one of k sampled solutions passes the tests. Execution matters because correct code can look nothing like a reference solution, so text similarity is the wrong yardstick.
GSM8K (Cobbe et al., 2021) tests multi-step arithmetic reasoning: grade-school math word problems requiring a short chain of steps. Scoring is exact match on the final numeric answer — which means a model can be right for the wrong reasons, and the harness’s answer-extraction logic becomes part of the measurement.
HELM (Liang et al., 2022) is not a single test but a multi-metric framework: many scenarios, each scored on several dimensions at once — accuracy, calibration, robustness, fairness, bias, toxicity, efficiency. Its core argument is the one this module keeps making: a single accuracy number hides the trade-offs you actually care about.
| Benchmark | What it probes | Metric | Blind spots |
|---|---|---|---|
MMLU | Broad knowledge and problem solving across 57 subjects, four-option multiple choice | Accuracy — did the model pick the labeled option? | Recognition, not generation: picking A–D says little about writing, explaining, or refusing. Highly sensitive to prompt format and shot count. Known to contain some flawed items. |
HumanEval | Code generation: 164 Python problems, function body from signature + docstring | pass@k — at least one of k samples passes hidden unit tests | Small task set, one language, short self-contained functions — far from real software work. Unit tests can pass wrong-in-spirit code. Sampling settings change the number. |
GSM8K | Grade-school multi-step math word problems | Exact match on the final numeric answer | Right answer via broken reasoning still scores. Answer extraction from free text is harness-sensitive. Ceiling on easy math says nothing about hard math. |
HELM | A framework: many scenarios × many metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) | One score per metric per scenario — deliberately not one number | Breadth over depth; aggregation and scenario selection are still editorial choices; running the full suite is expensive, so published slices vary. |
pass@k, precisely — and why temperature is part of the score
The naive way to compute pass@k — sample exactly k solutions per problem, check whether any passes — is an unbiased but noisy estimate. Chen et al. (2021) instead sample n ≥ k completions per problem, count the c that pass, and compute the probability that a random k-sized subset of the n samples would contain at least one passing solution:
pass@k = E[ 1 − C(n−c, k) / C(n, k) ]
where C(·,·) is the binomial coefficient and the expectation is over problems. Read it as: “out of all ways to draw k from my n samples, what fraction avoids drawing only failures?” — averaged across the task set.
The practical consequence engineers miss: pass@k depends on sampling settings, not just the model. Low Temperature concentrates probability on the model’s single best guess — good for pass@1, wasteful for pass@100, where you want diverse attempts. Higher temperature trades single-shot reliability for coverage across many samples. So a pass@k claim without the sampling configuration attached is an incomplete sentence — the same lesson as the harness point above, wearing a math costume.
Key terms: Benchmark, Eval harness, MMLU, HumanEval, pass@k, HELM
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.