Chinchilla: Compute-Optimal Training
Lesson 2 of 3 in Scaling Laws.
A training budget is one number — total FLOPs — but it buys two things: a bigger model, or more tokens through it. The approximation C ≈ 6·N·D (about 6 FLOPs per parameter per training token, forward plus backward; Kaplan et al. 2020, §2.1) makes the trade explicit: at fixed C, doubling N halves the D you can afford. Compute-optimal training is the question of where on that line the loss is lowest.
Kaplan’s fits had answered: mostly grow the model. In 2022, Hoffmann et al. at DeepMind re-ran the experiment — over 400 training runs, and, importantly, with the Learning rate schedule tuned to match each run’s token budget rather than one schedule for all — and got a different answer: N and D should grow roughly in step. By that analysis, the flagship models of the era were substantially undertrained: too many parameters, too few tokens.
Then they proved it the convincing way. Chinchilla, a 70B-parameter model trained on 1.4 trillion tokens, used roughly the same compute as its 280B sibling Gopher — a quarter of the parameters, about four times the data — and outperformed it across their evaluations (Hoffmann et al. 2022). Same bill, better model, and cheaper to run afterwards because it is smaller. Few papers have moved industry practice faster.
Line chart of loss versus model size for two fixed compute budgets, with model size on a log axis. Each curve is U-shaped: at 10^21 FLOPs loss falls from 2.97 at 30 million parameters to a minimum of 2.33 near 1.8 billion parameters, then rises to 2.57 at 30 billion. The 10^23 FLOPs curve sits lower, with its minimum of about 2.01 near 15 billion parameters. The chart shows that for a fixed budget there is an optimal model size, and that bigger budgets shift the optimum down and to the right.
The workhorse behind that figure is the paper’s parametric loss surface (App. D.2, Eq. 10):
L(N, D) = E + A/N^α + B/D^β, fitted as E = 1.69, A = 406.4, α = 0.34, B = 410.7, β = 0.28.
Read it as three stacked shortfalls. E = 1.69 is the floor: the irreducible entropy of the text itself, the loss a perfect model of infinite size and data would still pay, because language is genuinely uncertain. A/N^α is the finite-model penalty — shrinks as parameters grow. B/D^β is the finite-data penalty — shrinks as tokens grow. A model is compute-optimal when a budget’s two penalties are balanced at the margin; pouring everything into one term while the other stays large is how you waste a fortune. That was the era’s diagnosis in one line: giant N, starved D.
Chinchilla itself sits at 1.4T tokens for 70B parameters — 20 tokens per parameter — and that ratio hardened into the field’s favorite heuristic.
Deriving the optimum from the fit
Take the parametric loss and the compute constraint, and the optimum falls out with one derivative.
Setup. Fix the budget C and substitute the constraint D = C/(6N) into the fit:
L(N) = E + A·N^(−α) + B·(6/C)^β · N^β
The first variable term falls as N grows (bigger model), the second rises (fewer tokens through it) — that is the U-shape in the figure above.
Optimize. Set dL/dN = 0:
−α·A·N^(−α−1) + β·B·(6/C)^β · N^(β−1) = 0
Solving for N:
N* = [ (α·A)/(β·B) ]^(1/(α+β)) · (C/6)^(β/(α+β))
and D* = C/(6N*) follows. With the fitted α = 0.34, β = 0.28: the exponent β/(α+β) = 0.28/0.62 ≈ 0.45 for N*, and α/(α+β) ≈ 0.55 for D*. Both grow close to the square root of the budget — the “grow them in step” headline — versus Kaplan’s earlier allocation, which tilted much harder toward N.
Sanity check. For C = 10²¹ FLOPs the formula gives N* ≈ 1.8×10⁹ and D* ≈ 9×10¹⁰ — exactly where the low point of the plotted curve sits. Note the fitted ratio D*/N* ∝ C^((α−β)/(α+β)) = C^0.097 drifts upward with budget: near Chinchilla’s own scale it passes through ~20 tokens per parameter, which is where the heuristic comes from — and why it is only a heuristic. All constants inherit the caveats of lesson one: another tokenizer or dataset refits E, A, B, α, β.
| Question | Kaplan et al. 2020 | Hoffmann et al. 2022 (Chinchilla) |
|---|---|---|
Grow N or D? | Mostly N: train very large models on modest data, stop well before convergence | Both, roughly in step — near √C each under the parametric fit |
Verdict on era’s models | On track — bigger was better | Undertrained: too many parameters for the tokens they saw |
Key methodological difference | Learning-rate schedule not retuned per token budget (the follow-up’s diagnosis) | Schedule matched to each run’s token budget across 400+ runs |
Existence proof | GPT-3: 175B parameters, ~300B tokens (Brown et al. 2020) | Chinchilla: 70B on 1.4T tokens, ≈ Gopher’s (280B) compute, better evaluations |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.