Reading the Loss Curve
Lesson 3 of 3 in Objectives and the Loss Curve.
Every training run in the world is watched through the same chart. The y-axis is the training Loss — the per-token Cross-entropy from lesson one, usually in nats. The x-axis is progress, measured in tokens seen or optimizer steps, and almost always drawn on a log scale, because that is the scale on which the curve’s story is legible.
The healthy shape has two acts. Act one is a cliff: loss starts near ln(vocabulary size) — the surprise of pure uniform guessing over the Vocabulary — and plunges as the model soaks up cheap probability: which tokens are common, which follow which, the local grammar of the corpus. Act two is a grind: on a log-x plot the descent settles into something close to a straight line, shaving loss slowly and steadily across orders of magnitude of data. That near-straight line on log axes is no accident — loss falls as a power law in data and model size, which is the whole subject of the next module, Scaling laws.
Line chart with tokens seen on a logarithmic x-axis from 1 to 400 billion and training loss in nats per token on the y-axis from about 2 to 11. Three lines: a healthy curve falling steeply from 10.8 to about 2.6 and flattening gradually; a plateaued curve that stops improving around loss 5.7 early in training; and a spike-and-recovery curve that tracks the healthy one but jumps abruptly to 7.5 partway through before returning to trend.
The pathologies are what you actually watch for. A plateau — loss flattening far above where comparable runs landed — says learning has stalled: a Learning rate schedule gone wrong, a data problem, or a model simply too small for the target. A Loss spike — a sudden vertical jump — is the signature of training instability: an unlucky batch meeting aggressive optimization and low-precision numerics (BF16 (bfloat16), Mixed precision). Some spikes recover on their own; some send the run back to an earlier Checkpoint. The OPT team published their logbook precisely so the world could see how often a real 175B-parameter run hit these — restarts and all (Zhang et al. 2022). The numerics behind spikes, and the recovery playbook, get their own module: Mixed Precision and Stability.
The second line on the chart is validation loss: the same cross-entropy, measured on held-out data the optimizer never touches. In classic machine learning you expect the two to diverge — training loss falling while validation rises means memorization. In LLM pre-training the regime is unusual: with a corpus so large that most runs make about one Epoch over it, nearly every batch is fresh, so train and validation typically track each other closely. The gap becomes a warning exactly when that assumption breaks — when data repeats, because you ran multiple epochs over a small corpus or because Deduplication missed heavy duplication. Duplicated text measurably promotes memorization (Lee et al. 2021), and a widening gap is how it shows up on the dashboard.
Loss starts near ln(vocab size) and never falls
The starting value itself is a diagnostic: uniform guessing over a vocabulary of size V costs −log(1/V) = ln V — about 10.8 nats for a 50k vocabulary. If loss sits there, the model is learning nothing: suspect the wiring before the model — data pipeline emitting garbage, labels not actually shifted, learning rate effectively zero.
Falls fast, then flattens far above comparable runs
A plateau. Check the learning-rate schedule first (a decay that kicked in too early is a classic), then data quality and repetition, then whether the model is simply too small for the loss you are targeting — the Scaling laws module shows how to predict where a given size should land.
A sudden vertical spike
A Loss spike. Watch it: transient spikes often recover within a few hundred steps. If loss diverges or plateaus higher than before, the standard operational answer is to restart from a recent Checkpoint, often with an adjustment — a lower learning rate, or skipping the data region that triggered it. Public logbooks like OPT’s record exactly this playbook in the wild (Zhang et al. 2022).
Training keeps falling, validation turns upward
Memorization. The optimizer is polishing performance on text it has already seen at the expense of text it has not. Look for repeated data: an extra Epoch over a small corpus, or duplication that survived cleaning. Trust the validation line — it is the one measuring what you actually want.
Loss looks great, downstream ability does not move
Loss is necessary, not sufficient. It measures average predictive surprise on the training distribution — not coding skill, not factual reliability. Two runs with near-identical loss can differ meaningfully on capabilities, and a Data mixture that underweights a skill will not teach it however pretty the curve. This is why the Evaluation domain exists.
Where should the curve end — and why never zero
The loss floor is not zero, and published work has even put numbers on it. The Chinchilla analysis fit final pre-training loss with a three-term function of model size N (parameters) and data D (tokens):
L(N, D) = E + A/N^α + B/D^β with fitted values E = 1.69, A = 406.4, α = 0.34, B = 410.7, β = 0.28 (Hoffmann et al. 2022, Appendix D.2, Eq. 10).
Read it right to left: one term falls as you add data, one falls as you add parameters, and E — the irreducible term — falls with nothing. It is the fitted estimate, in that paper’s setup, of the entropy of the text itself: the loss a hypothetical infinitely large model trained on infinite data would still pay, because language is genuinely uncertain. Every training curve is a descent toward a floor it can approach but never touch.
Two honesty caveats before you quote numbers. The absolute values depend on the tokenizer and the training distribution — a different vocabulary slices the same text into different tokens, and per-token loss changes with it — so the shapes and ratios generalize, the constants do not. And the fit describes final loss at the end of well-tuned runs, not every wobble along the way. You can explore how the three terms trade off interactively in the Scaling Law Plotter, and the next module makes this equation the main character.
In production
The loss curve is the training-run dashboard. Whatever the platform, the same telemetry loop exists: the training job emits per-step metrics, a dashboard renders them live, and checkpoints plus alerts turn a bad curve into a recoverable event instead of a lost month of cluster time.
AWS
Managed training jobs on AWS stream metrics from the training containers into live monitoring dashboards and logs, with TensorBoard-style visualization for per-step loss. The operational pattern the platform is built around: log loss every few steps, write checkpoints to object storage on a schedule, and rely on automatic job resumption after hardware faults — so a spike or a failed node means "resume from the last good checkpoint", not "start over". Alarms on stalled or diverging metrics are what page a human before a week of compute burns on a broken run.
Azure
Azure Machine Learning treats a training run as a tracked experiment: metrics logged from the training script (MLflow-compatible) are stored per run and charted live, so the loss curve, learning-rate schedule, and throughput sit side by side in the run view. Because runs are first-class objects, comparing the current curve against previous runs of the same recipe is built in — which is exactly how plateaus and regressions are caught: not by staring at one line, but by overlaying it on the last known-good one.
Google Cloud
Vertex AI training integrates with a managed TensorBoard service: the job writes event logs, the dashboard renders loss and validation curves as they stream in, and checkpoints land in cloud storage for restart. The habit the tooling encourages is the right one everywhere: decide before launch what curve you expect (starting value near ln of vocabulary size, target region from scaling-law estimates), then treat deviations from that expectation — not absolute numbers — as the alert condition.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.