The Discovery: Loss Follows a Power Law
Lesson 1 of 3 in Scaling Laws.
Before 2020, “make the model bigger” was a hunch backed by anecdotes. Then Kaplan et al. trained families of Transformer language models across a huge range of sizes and measured what happened to the Cross-entropy Loss. The result — Scaling Laws for Neural Language Models (arXiv:2001.08361) — turned the hunch into an engineering discipline.
The finding: hold everything else generous, vary one budget, and loss falls as a power law in that budget. Three versions of the same shape:
- Model size. With enough data, loss falls as a power law in the number of non-embedding parameters N.
- Data. With a large enough model, loss falls as a power law in the number of training tokens D.
- Compute. With both allocated well, loss falls as a power law in total training compute C.
A power law L = (X_c / X)^α has a signature you can spot from across the room: plot loss against the budget on log-log axes and you get a straight line, with slope −α. Kaplan et al. observed these lines holding, in their words, with some trends spanning more than seven orders of magnitude — and no wall in sight. The same paper found that, within the transformer family, architectural details like depth versus width mattered far less than raw scale: where you spend matters less than how much you spend.
Log-log line chart of loss versus number of non-embedding parameters, from one million to one trillion. The curve is a straight descending line: loss 4.02 at 10^6 parameters, 2.83 at 10^8, 1.99 at 10^10, and 1.41 at 10^12, illustrating a power law with exponent 0.076.
Look at what that line buys you. Predictability: train a family of small, cheap models, fit the line, and extrapolate to forecast the loss of a run a thousand times bigger — before committing the budget. Frontier training runs are planned this way; the fit is the business case.
Now look at what the exponent costs you. α_N ≈ 0.076 means every 10× in parameters multiplies the size-limited loss by 10^−0.076 ≈ 0.84 — a 16% shave. Halving that loss term takes roughly a 10,000-fold increase in N (2^(1/0.076) ≈ 9,000). Scaling laws are simultaneously the best news in the field — no wall — and a brutal statement about diminishing returns: progress is reliable and exponentially expensive.
The paper drew one more conclusion that set the agenda for two years: under its fits, a fixed compute budget was best spent mostly on model size, training very large models on comparatively modest data and stopping before convergence. The industry listened. GPT-3 arrived within months — 175B parameters trained on roughly 300B tokens (Brown et al. 2020) — and the bigger-is-better era was on.
| Budget | Exponent α | Constant X_c | What the law says |
|---|---|---|---|
Model size N | 0.076 | 8.8×10¹³ non-embedding parameters | With enough data, bigger models reach lower loss — slowly: 10× the parameters buys ≈16% off the size-limited loss |
Dataset size D | 0.095 | 5.4×10¹³ tokens | With a big enough model, more tokens buy lower loss — a slightly steeper line than the one for N |
Compute C | 0.050 | 3.1×10⁸ PF-days | With compute allocated well between N and D, total training FLOPs predict loss — the planning curve for a run |
The era the straight lines launched
- 2019-02-14 — GPT-2 and staged release:
A 1.5B-parameter GPT trained on web text showed surprising zero-shot ability across tasks, and OpenAI staged its release over misuse concerns. It was the first clear signal that scaling one recipe broadens capability — and the first mainstream release-policy debate.
- 2020-01-01 — Scaling laws for neural LMs:
Kaplan et al. measured loss falling as smooth power laws in parameters, data, and compute, with architectural details mattering far less. It turned "make it bigger" from a hunch into a predictive engineering discipline.
- 2020-05-01 — GPT-3: in-context learning:
At 175B parameters, GPT-3 exposed a new interface: in-context learning, where the model performs tasks from instructions and a few examples in the prompt, with no gradient updates. Prompting began to replace fine-tuning as the primary way to program a model.
- 2022-03-01 — Chinchilla: compute-optimal training:
Hoffmann et al. showed most large models were undertrained: at a fixed compute budget, parameters and training tokens should grow roughly in step (about 20 tokens per parameter). It reset scaling strategy toward smaller models trained on far more data.
- 2022-11-30 — ChatGPT launches:
A free chat interface over an RLHF-tuned GPT-3.5 model reached an estimated 100 million users within two months. It was the moment LLMs became a consumer product, and it reset the industry’s research and deployment agenda around assistants.
- 2023-02-01 — LLaMA: the open-weights era begins:
Meta trained 7B–65B models Chinchilla-style on trillions of tokens and released the weights to researchers; the 13B model rivaled GPT-3. The weights spread far beyond the research program and ignited the open-weights ecosystem of fine-tunes and local inference.
- 2023-03-14 — GPT-4:
A multimodal model (text and image input) with large capability jumps — top-decile bar-exam performance among them — whose technical report disclosed neither architecture nor training data. It marked both the new capability frontier and the industry’s turn away from openness.
Key terms: Scaling laws, Compute-optimal, FLOPs, Loss, Parameter
Tool: Scaling Law Plotter — Plot these laws yourself: the Scaling Law Plotter draws the published Kaplan and Chinchilla fits, lets you move the budget sliders, and shows what the extrapolations do and do not promise.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.