After Chinchilla: Overtraining on Purpose
Lesson 3 of 3 in Scaling Laws.
Compute-optimal answers a narrow question: given one pot of training FLOPs, what is the lowest loss it can buy? That is the right question for a research lab racing a benchmark. It is the wrong question for a model you intend to serve.
Training is spent once. Inference is paid on every token, for the life of the model — and each generated token costs FLOPs and memory (weights plus KV cache) roughly proportional to model size. A model with a quarter of the parameters is, to first order, a quarter of the serving compute per token: more requests per GPU, lower latency, and hardware small enough to sit on cheaper machines — or on a phone. So a deployed model’s true cost curve is training (once) + inference (per token × tokens served), and for a popular product the second term dwarfs the first.
That changes the optimization. Take the training budget you were going to spend compute-optimally and instead push it through a smaller model as extra data. The loss you reach is slightly worse than the compute-optimal point for that budget — you are past the U-curve’s minimum — but every one of the trillions of tokens the model will later serve is now dramatically cheaper. This is called overtraining, and post-Chinchilla it is standard practice: not a mistake, an inference-economics decision.
Line chart of loss versus training tokens for a fixed 7-billion-parameter model, tokens on a log axis from 140 billion to 8 trillion. Loss falls from 2.18 at 140 billion tokens to 2.05 at 1 trillion and 1.97 at 8 trillion, flattening toward a floor near 1.87. The curve shows diminishing but never-zero returns from training a small model far past the compute-optimal token count.
The paper that made this strategy famous said it out loud. Meta’s LLaMA (Touvron et al. 2023, arXiv:2302.13971) set its objective as the best possible performance at various inference budgets — and trained its models on a trillion tokens and more. The 7B model’s roughly 1T tokens is about 140 tokens per parameter, seven times the Chinchilla heuristic. Deliberately compute-suboptimal per training FLOP; deliberately excellent per serving dollar. The open-weights models that followed pushed the same logic further, and small-but-heavily-overtrained is now the default recipe for models meant to be run widely.
Cloud architects will recognize the shape of this argument instantly: it is capex versus opex. The training run is capital expenditure — painful, one-time, on the lab’s bill. Serving is operating expenditure — perpetual, scaling with traffic, often on your bill. Chinchilla optimized the capex. The industry, rationally, optimizes the sum — and for anything with users, the sum is dominated by opex.
| Dimension | Compute-optimal (Chinchilla-style) | Overtrained smaller model (LLaMA-style) |
|---|---|---|
What it minimizes | Loss per unit of training compute | Cost per unit of quality served over the model’s life |
Shape of the run | N and D balanced near the fit’s optimum (~20 tokens/parameter at Chinchilla’s scale) | Smaller N, D pushed far past the optimum — 100+ tokens/parameter is common |
End-of-training loss | The lowest that budget can buy | Slightly worse — you are past the U-curve minimum on purpose |
Serving profile | Bigger model: more FLOPs and memory per token, fewer requests per accelerator | Smaller model: cheaper tokens, lower latency, fits smaller (even edge) hardware |
When it wins | One-shot research results; models that will rarely be served | Products with sustained traffic — lifetime inference tokens ≫ training tokens |
In production
You may never train a model, but you inherit this trade-off every time you pick one: on managed platforms, model size is the knob behind per-token price, latency, and the hardware your traffic needs.
AWS
On Amazon Bedrock the menu of model sizes is the scaling-laws trade-off, surfaced as price and latency tiers: smaller variants of a family exist because someone overtrained them so they would serve cheaply. Sizing guidance is the same as the decision rule above — estimate lifetime tokens through the system, then test whether the smaller, heavily-trained model clears your quality bar, because every size step you avoid is paid back on every token. Self-hosting on EC2 GPU instances makes it sharper still: parameter count decides how much accelerator memory the weights and KV cache occupy, hence how many replicas fit per node and what your fleet costs at peak.
Azure
In Azure AI Foundry the same logic governs deployment planning: throughput quotas and provisioned capacity are consumed faster by larger models, so a smaller overtrained model stretches the same provisioned throughput across more requests. The model catalog’s small/medium/large variants within a family are the overtraining strategy productized — evaluate down the size ladder with your own evals, and reserve the biggest model for the traffic slice that measurably needs it, not as the default.
Google Cloud
On Vertex AI, size tiers within a model family (and the option to serve open-weights models on your own GPU or TPU capacity) expose the identical economics: fewer parameters means fewer accelerator-FLOPs and less memory per token, so more of your quota becomes throughput. The evergreen mechanism to remember on any cloud: training cost was someone’s one-time capex; your serving bill is opex on every token — so the model-size decision is an inference-economics decision, and scaling laws are why the small-but-overtrained option exists at all.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.