Pre-training

Data pipelines, objectives, scaling laws, and the engineering of a training run

• Data Pipelines and Curation — Where training text actually comes from, and the pipeline that turns a crawl into a corpus: filtering, deduplication, and the mixture decisions that shape a model. (4 lessons, 45 min)
• Objectives and the Loss Curve — Causal language modeling and its rivals — masked and span objectives — plus how to read the one chart every training run lives by. (3 lessons, 40 min)
• Scaling Laws — The discovery that loss follows the budget: Kaplan’s power laws, Chinchilla’s compute-optimal correction, and why modern models overtrain on purpose. (3 lessons, 45 min)
• Compute Budgets — From FLOPs accounting to GPU-hours: how training budgets are actually estimated, and where the money goes. (3 lessons, 40 min)
• Distributed Training — No single device fits the model or the data: data, tensor, and pipeline parallelism — and how real runs combine all three. (5 lessons, 50 min)
• Mixed Precision and Stability — FP32, FP16, BF16 — why training runs in mixed precision, and what loss spikes, checkpoints, and restarts look like at scale. (3 lessons, 40 min)
• In Production: Should You Ever Pre-train? — The domain’s production capstone: the honest decision framework for training from scratch versus continued pre-training versus not doing this at all — and what a training cluster demands. (3 lessons, 35 min)