Cleaning and Filtering
Lesson 2 of 4 in Data Pipelines and Curation.
Extraction gets you text; filtering decides which text deserves compute. Two philosophies dominate, and most real pipelines use both.
Heuristic filters are cheap, transparent rules. C4 is the best-documented example because Raffel et al. (2019) published theirs: keep only lines ending in terminal punctuation, drop pages containing words from a profanity blocklist, drop anything containing the placeholder string “lorem ipsum”, drop pages containing curly braces (a blunt way to exclude code). Each rule is defensible in isolation — and each has a shadow. The curly-brace rule is the famous cautionary tale: it makes C4 nearly code-free, which is exactly what you would not want for a model that should program. The filter did not malfunction; it did precisely what it was told, and the corpus quietly inherited the decision.
Classifier filters replace hand rules with a learned judgment: train a small model to score documents by “quality”, where quality is operationally defined as resembling some reference corpus — pages linked from curated sources, or text that looks like edited reference writing. This scales past what rules can express, but the definition is the catch: the classifier promotes whatever resembles its reference and demotes everything else — dialects, informal registers, transcribed speech, niche technical writing. A quality filter is an opinion about language, executed at corpus scale.
| Stage | How it works | What it removes | What it risks |
|---|---|---|---|
Boilerplate removal | Rules or learned extractors isolate the main text of a page | Menus, footers, cookie banners, ad copy | Mangled extraction on unusual layouts — broken text becomes training data |
Language ID | A fast classifier labels each document’s language | Documents outside the target language set | Misclassifying low-resource languages and code-switched text — they vanish from the corpus |
Quality heuristics | Hand rules: punctuation, length, symbol ratios, blocklists | Spam, placeholder text, machine-generated junk | Blunt rules with side effects — C4’s curly-brace rule also removed code |
Classifier quality filter | A learned model scores similarity to a “high-quality” reference | Documents scored unlike the reference corpus | Inherits the reference’s biases; demotes dialects and informal registers wholesale |
Toxicity filtering | Blocklists or classifiers flag harmful content | Toxic, explicit, or hateful documents | Over-removal of discussions about harm (medical, legal, support communities) |
PII scrubbing | Pattern matchers and NER-style detectors find personal data | Emails, phone numbers, addresses, IDs | Imperfect recall — some PII survives into weights; precision errors corrupt legitimate text |
One habit makes you a better reader of every dataset paper: for each filter, ask what it removes, and then ask what else it removes. The first answer is in the paper’s motivation; the second is usually discovered later, by someone probing the model. Filters compose, too — a document must survive every stage, so individually mild filters can stack into a narrow corpus. This is why serious pipelines publish their rules (as C4 did) or their source composition (as The Pile did): the filter list is part of the model’s specification, as much as the Parameter count is.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.