Objectives and the Loss Curve

Causal language modeling and its rivals — masked and span objectives — plus how to read the one chart every training run lives by.

Content last verified 2026-09.

Lessons

1. Causal Language Modeling
2. Masked, Span, and the Road Not Taken
3. Reading the Loss Curve

Sources

• Radford et al. (2019) — Language Models are Unsupervised Multitask Learners (GPT-2, causal LM at scale)
• Devlin et al. (2018) — BERT: Pre-training of Deep Bidirectional Transformers (masked language modeling)
• Raffel et al. (2019) — Exploring the Limits of Transfer Learning with T5 (span corruption, C4)
• Zhang et al. (2022) — OPT: Open Pre-trained Transformer Language Models (public logbook of a real run)
• Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (the parametric loss fit)