The Mixture: What the Model Eats

Lesson 4 of 4 in Data Pipelines and Curation.

A corpus is not a pile — it is a recipe. Once the sources are cleaned and deduplicated, someone must decide the mixture: what fraction of the training tokens come from each source. The web crawl is huge but noisy; books and reference text are scarce but dense. So pipelines do not sample sources in proportion to their raw size — they weight them, typically upsampling small high-value sources and letting the crawl fill the bulk. Llama 1 is a usefully transparent example: Touvron et al. (2023) published the exact sampling proportion for every source in the mixture, along with how many times each source was passed over during training.

That last detail is the second dial: epochs versus fresh tokens. When a source is small and valuable, you can show it to the model more than once. The evidence generally favors fresh text where you can get it — lesson three showed why re-reading buys little — and follow-up research on data-constrained training suggests a few repetitions of good sources cost little, with returns decaying as repetitions mount. Treat any precise repeat-count rule as an empirical claim to check against current literature, not a law.

Key terms: Data mixture, Deduplication, Epoch, Token, Knowledge cutoff

Horizontal bar chart of an invented training data mixture: filtered web crawl 60%, code 15%, books 8%, academic papers 7%, reference text 5%, dialogue and forums 5%. Illustrates that a web crawl typically supplies most tokens while curated sources get deliberate weight.

A toy pre-training mixture, invented for teaching: the web crawl dominates by volume while smaller curated sources are deliberately upweighted. Real mixtures are design secrets more often than not — Llama 1 (Touvron et al. 2023) is a rare fully documented example. (illustrative — source: Touvron et al. (2023) — LLaMA, a documented real mixture)

Here is the part that makes the mixture more than an engineering detail: it is a product decision. Double the code fraction and you are building a better coding model at some cost to prose; add heavy multilingual weight and you are choosing markets; upweight academic text and you shape the register the model reaches for. Teams building code assistants and teams building writing assistants want different corpora, and no amount of downstream tuning fully substitutes for what pre-training exposure builds. When a model family ships a variant that is unexpectedly strong at some domain, the uninteresting-sounding explanation — the mixture was different — is usually the right one.

Two boundary notes. Whether benchmark test data leaked into the mixture — and how to detect it after the fact — is the contamination story, told properly in the Evaluation domain. And sizing the total token budget that the mixture must fill is the scaling-laws story: the Scaling Law Plotter lets you play with the published fits directly.

In production

You may never assemble a pre-training corpus, but its consequences reach your procurement checklist: enterprises now routinely ask what a model was trained on, under what rights, and who carries the risk. The mechanisms below are evergreen; the legal analysis itself belongs to our sister AI Governance Academy.

AWS

On Amazon Bedrock, each hosted model comes from a named provider under that provider’s own terms — so training-data provenance questions flow through to the provider’s documentation and end-user license agreement. Procurement reviews typically compare providers’ model documentation on training-data sourcing, and the separation of concerns matters: data you bring for customization stays yours under the service’s data-use commitments, distinct from whatever corpus pre-trained the base model.

Azure

Microsoft surfaces provenance through the model catalog: models in Azure AI Foundry carry provider documentation and transparency notes, and enterprise agreements spell out who indemnifies what for model outputs. The procurement-relevant mechanism is the paper trail — model cards and responsible-AI documentation describing training-data sourcing at whatever level of detail the provider discloses, which for many frontier models remains coarse.

Google Cloud

Google’s Vertex AI Model Garden likewise attaches model cards and provider terms to each model, and Google publishes documentation on data governance for its generative services — covering how customer data is and is not used for training. The evergreen questions are the same on all three clouds: what corpus, whose rights, what disclosure, and who bears the risk — with the honest answer often being that public disclosure stops at the mixture’s broad strokes.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.