Where the Data Comes From

Lesson 1 of 4 in Data Pipelines and Curation.

A Base model knows nothing except what its training corpus taught it. Every fact it can recall, every language it speaks, every API it can call from memory — all of it entered through the data pipeline. That makes the corpus the single highest-leverage design artifact in pre-training, and it is assembled, not found.

The backbone is almost always the web. The canonical public example is Common Crawl — a nonprofit that has been publishing an open corpus of the public web since 2007 and publishing the results openly, snapshot after snapshot. What a crawl delivers is not clean prose: it is raw HTML — navigation menus, cookie banners, ads, templates, spam, and the same page mirrored under a dozen URLs — with the actual text buried inside. Turning that into training text is the whole story of this module. C4 (the Colossal Clean Crawled Corpus) is the classic worked example: Raffel et al. (2019) started from a Common Crawl snapshot and published exactly which cleaning rules produced the corpus that trained T5.

Around the web backbone, curators layer deliberate sources, each bought for a reason: code from public repositories (models learn programming — and, many researchers argue, some structured reasoning — from it), books (long-range coherent prose that teaches a model to hold a thread far beyond a web page), reference text like Wikipedia (dense, factual, comparatively clean), and academic and technical writing (papers, standards, Q&A forums). The Pile (Gao et al. 2020) is the landmark of this philosophy: 22 distinct sources — from arXiv to GitHub to legal text — deliberately combined into one 800GB-scale corpus on the bet that diversity itself improves the model.

From raw crawl to training corpus

  1. Raw web crawl

    A Common Crawl-style snapshot: raw HTML plus metadata for billions of pages. Boilerplate, ads, spam, and duplicates included.

  2. Text extraction

    Strip markup and boilerplate (menus, footers, cookie banners) to recover the main text of each page. Imperfect by nature — extraction errors become training text.

  3. Language identification

    Classify each document’s language and keep the ones the corpus is meant to cover. The first big fork in what the model will speak. (Lesson 2.)

  4. Quality + safety filtering

    Heuristics and classifiers drop low-quality, toxic, or unwanted documents; PII gets scrubbed. Every rule is a trade-off. (Lesson 2.)

  5. Deduplication

    Remove exact and near-duplicate text so the model does not study the same page a thousand times. (Lesson 3.)

  6. Mixing with curated sources

    Weight the cleaned web against code, books, and reference text — the mixture decision. (Lesson 4.)

  7. Training corpus (tokens)

    Tokenized, shuffled, and sharded for the training run. From here on, the model’s world is fixed.

How big are these corpora? Resist the urge to answer in gigabytes. The unit that matters is the Token, because tokens are what the model actually consumes: the Loss is per token, the training compute bill scales with token count, and the Scaling laws that set data budgets are written in tokens. A few documented anchors: GPT-3 was trained on roughly 300 billion tokens (Brown et al. 2020); Chinchilla trained a 70B-parameter model on 1.4 trillion tokens (Hoffmann et al. 2022); Llama 1 trained on over 1 trillion tokens of publicly sourced data (Touvron et al. 2023). The direction of travel since has been more tokens per parameter, for reasons the scaling-laws module makes precise.

Two intuitions to carry out of this lesson. First, the pipeline is a funnel: a raw crawl is vastly larger than the corpus that survives extraction, filtering, and deduplication — most of what is crawled never reaches the model. Second, the corpus is the model’s entire world: whatever the crawl date was becomes the model’s Knowledge cutoff, whatever languages survived filtering are the languages it speaks, and whatever the mixture under-weighted, the model will be weak at. When a model surprises you, the explanation is very often upstream, in decisions someone made here.

The scale era: data milestones alongside model milestones

  • 2018-02-01 — ELMo: contextual representations:

    Peters et al. derived word representations from a pretrained bidirectional LSTM language model, so a word’s vector depends on its sentence. It showed that pretraining on raw text transfers broadly across tasks — the step just before whole-model fine-tuning.

  • 2018-06-01 — GPT-1: generative pre-training:

    OpenAI pretrained a decoder-only transformer as a language model on unlabeled text, then fine-tuned it for downstream tasks. It fixed the architecture family — the decoder-only transformer LM — that GPT-2, GPT-3, and GPT-4 would scale.

  • 2018-10-01 — BERT:

    Devlin et al. pretrained a bidirectional transformer encoder with masked-language modeling and released the weights; fine-tuned BERT swept the NLP benchmarks of its day. It made "download a pretrained transformer and fine-tune it" the default workflow.

  • 2019-02-14 — GPT-2 and staged release:

    A 1.5B-parameter GPT trained on web text showed surprising zero-shot ability across tasks, and OpenAI staged its release over misuse concerns. It was the first clear signal that scaling one recipe broadens capability — and the first mainstream release-policy debate.

  • 2020-01-01 — Scaling laws for neural LMs:

    Kaplan et al. measured loss falling as smooth power laws in parameters, data, and compute, with architectural details mattering far less. It turned "make it bigger" from a hunch into a predictive engineering discipline.

  • 2020-05-01 — GPT-3: in-context learning:

    At 175B parameters, GPT-3 exposed a new interface: in-context learning, where the model performs tasks from instructions and a few examples in the prompt, with no gradient updates. Prompting began to replace fine-tuning as the primary way to program a model.

  • 2022-03-01 — Chinchilla: compute-optimal training:

    Hoffmann et al. showed most large models were undertrained: at a fixed compute budget, parameters and training tokens should grow roughly in step (about 20 tokens per parameter). It reset scaling strategy toward smaller models trained on far more data.

  • 2023-02-01 — LLaMA: the open-weights era begins:

    Meta trained 7B–65B models Chinchilla-style on trillions of tokens and released the weights to researchers; the 13B model rivaled GPT-3. The weights spread far beyond the research program and ignited the open-weights ecosystem of fine-tunes and local inference.

  • 2023-03-14 — GPT-4:

    A multimodal model (text and image input) with large capability jumps — top-decile bar-exam performance among them — whose technical report disclosed neither architecture nor training data. It marked both the new capability frontier and the industry’s turn away from openness.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.