Deduplication: The Unglamorous Win
Lesson 3 of 4 in Data Pipelines and Curation.
The web repeats itself, relentlessly: mirrors, syndicated articles, templated pages, boilerplate pasted across thousands of sites. Left in the corpus, that repetition costs you three ways — and the systematic study is Lee et al. (2021), Deduplicating Training Data Makes Language Models Better, which is worth reading in full.
Memorization. Text a model sees many times is text it learns to reproduce verbatim. Lee et al. found that models trained on deduplicated data emit memorized training text roughly ten times less often — directly relevant if your training data contains anything you would not want recited back (personal data, licensed text).
Wasted compute. Every Token in the budget costs the same to train on, and the data budget is finite. The thousandth copy of the same passage teaches almost nothing new; those FLOPs could have bought novel text. Lee et al. report equal or better Perplexity with fewer training steps after deduplication — the rare free lunch.
Evaluation contamination. Duplication does not respect your train/test split: popular text ends up in both the crawl and the benchmarks, and a model that has seen the test set posts scores it has not earned. Deduplication against evaluation sets is the first line of defense.
There are two levers, and serious pipelines pull both: exact matching — hash whole documents, or find long verbatim spans shared across documents — and fuzzy (near-duplicate) matching, MinHash-style similarity that catches documents which are almost the same without being byte-identical. Sort the cases yourself:
Interactive sorting exercise: Each card is a kind of duplicate found in real crawls. Sort it by the cheapest technique that reliably catches it.
How MinHash finds near-duplicates without comparing every pair
The naive plan — compare every document with every other — dies at web scale: a billion documents means ~10¹⁸ pairs. MinHash is the trick that makes near-duplicate detection affordable, and it is built from three moves.
Move 1: turn documents into sets. Break each document into overlapping word n-grams (“shingles”) — say every 5-word window. Two documents are near-duplicates when their shingle sets overlap heavily, measured by Jaccard similarity: the size of the intersection over the size of the union.
Move 2: compress each set into a short signature. Apply a hash function to every shingle in a document and keep only the minimum hash value. Here is the elegant part: for a random hash function, the probability that two documents share the same minimum is exactly their Jaccard similarity — the shingle that hashes lowest across both sets is equally likely to be any shingle in the union, and it produces a matching minimum precisely when it lies in the intersection. Use, say, 128 different hash functions and you get a 128-number signature per document; the fraction of matching positions between two signatures estimates their similarity. Million-shingle documents become fixed-size fingerprints.
Move 3: find candidate pairs without all-pairs comparison. Split each signature into bands of a few numbers each, and bucket documents by each band’s value (locality-sensitive hashing). Highly similar documents almost surely collide in at least one band; dissimilar ones almost never do. Only the colliding candidates get checked properly, then duplicate clusters are collapsed to a single kept copy.
None of this is exotic — it is 1990s-era search-engine machinery. What changed is the stakes: at pre-training scale, this bookkeeping decides what a multi-million-dollar run actually learns.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.