Scaling Laws

The discovery that loss follows the budget: Kaplan’s power laws, Chinchilla’s compute-optimal correction, and why modern models overtrain on purpose.

Content last verified 2026-09.

Lessons

1. The Discovery: Loss Follows a Power Law
2. Chinchilla: Compute-Optimal Training
3. After Chinchilla: Overtraining on Purpose

Sources

• Kaplan et al. (2020) — Scaling Laws for Neural Language Models
• Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (Chinchilla)
• Touvron et al. (2023) — LLaMA: Open and Efficient Foundation Language Models
• Brown et al. (2020) — Language Models are Few-Shot Learners (GPT-3)