Benchmarks and Their Limits

What a benchmark actually measures, how benchmarks saturate and get gamed, and how to read a leaderboard number without being fooled.

Content last verified 2026-09.

Lessons

1. What a Benchmark Is
2. Saturation: When the Ceiling Arrives
3. Gaming, Goodhart, and Leaderboards
4. Reading a Benchmark Number

Sources

• Hendrycks et al. (2020) — Measuring Massive Multitask Language Understanding (MMLU)
• Chen et al. (2021) — Evaluating Large Language Models Trained on Code (HumanEval, pass@k)
• Cobbe et al. (2021) — Training Verifiers to Solve Math Word Problems (GSM8K)
• Liang et al. (2022) — Holistic Evaluation of Language Models (HELM)