Data Pipelines and Curation

Where training text actually comes from, and the pipeline that turns a crawl into a corpus: filtering, deduplication, and the mixture decisions that shape a model.

Content last verified 2026-09.

Lessons

1. Where the Data Comes From
2. Cleaning and Filtering
3. Deduplication: The Unglamorous Win
4. The Mixture: What the Model Eats

Sources

• Raffel et al. (2019) — Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (C4)
• Gao et al. (2020) — The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Lee et al. (2021) — Deduplicating Training Data Makes Language Models Better
• Touvron et al. (2023) — LLaMA: Open and Efficient Foundation Language Models