The Problem: Memorized Exams

Lesson 1 of 3 in Data Contamination.

Every Benchmark rests on one assumption: the model has never seen the test items. That is what a Held-out set means — held out. The score is a proxy for how the model handles problems it was not trained on, and every claim built on top of the score — this model is better at reasoning, that one at code — inherits the assumption.

Web-scale training breaks it quietly. Modern LLMs train on corpora harvested from the public web, and benchmarks live on that same web. Test sets are published openly so researchers can run them — HumanEval’s problems ship inside its GitHub repository, and MMLU’s official repository links the full test set for anyone to download. Papers quote items. Blog posts walk through examples with solutions. Tutorial notebooks embed whole datasets. Q&A threads dissect individual questions. None of this is misconduct; it is how open science works. But a crawler collecting pages by the billion has no idea it just swallowed an exam, and the model that trains on the crawl gets to study the test before sitting it.

The result is a score that measures the wrong thing. A model that trained on the test items does not need the capability the benchmark was built to proxy — it needs memory. And the two are indistinguishable on the scoreboard: Contamination and genuine capability produce exactly the same number.

The leak path: how a test item ends up in training data

  1. Benchmark released

    The test set is published — typically in a public repo — so that anyone can evaluate on it. Openness is the point, and the vulnerability.

  2. Copies spread across the web

    Mirrors, forks, papers quoting items, blog walkthroughs with answers, tutorial notebooks, forum threads. Each copy is one more page a crawler can find.

  3. Web crawl ingests the pages

    Crawls collect pages wholesale. Nothing marks a page as “this is a test set” — an exam looks like any other well-structured text.

  4. Filtering fails to catch it

    Quality filters keep clean, well-formatted text — benchmark items score well. Deduplication removes repeated copies at best, not the first one.

  5. Model trains on the items

    The test items are now ordinary training tokens. The model may memorize them verbatim or absorb them more diffusely.

  6. Model is evaluated on the same benchmark

    The evaluator, often a different team years later, has no idea the items were inside the corpus.

  7. Inflated score

    The number reports recall of seen items as if it were capability on unseen ones. From the outside, nothing looks wrong.

This problem is exactly as old as web-scale training, and the record proves it. The GPT-3 paper (Brown et al., 2020) shipped with its own contamination analysis: the authors searched their training data for overlaps with the benchmarks they reported, flagged the affected test sets, and published scores on cleaned subsets alongside the headline numbers — candidly noting that a bug in their filtering had let some overlaps through and was found too late to retrain. A year later, Dodge et al. (2021) audited C4 — a widely used public corpus distilled from Common Crawl — and found test examples from NLP benchmarks sitting inside it.

Read those two papers together and the lesson is structural: contamination is not a story about one careless lab. It is the default state of any corpus scraped from the same web where benchmarks are published, and the honest response — then and now — is to detect, disclose, and discount, not to promise it never happens.

Key terms: Contamination, Benchmark, Held-out set, Golden set, Deduplication

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.