Being Wrong, Measurably: Loss
Lesson 3 of 4 in Next-Token Prediction, Really.
The guessing game has a property that makes it perfect for training machines: it grades itself. Take any sentence ever written, cover the next token, let the model predict, then uncover. The right answer was sitting in the text all along. No human labeling, no answer key to build — every position in every document is a question with a known answer. This is why the game, and not something nobler, became the training objective of every Large language model (LLM): it turns the entire written internet into an exam with the answers attached.
But ‘the model got it wrong’ is not a number, and training needs a number. Here is the one the field settled on. When the true next token is revealed, look up the probability the model had assigned to that specific token, and take its negative logarithm: the loss is −log p(true token). You can read this quantity as surprise. Assign the truth a high probability and −log p is small — you saw it coming, low surprise, low Loss. Assign it a tiny probability and −log p explodes — you were confidently looking elsewhere, and the loss says so. This measure is called Cross-entropy, and its shape encodes a value judgment worth noticing: it does not ask were you right, it asks how much probability did you put on the truth. Being uncertain costs a little. Being confidently wrong costs enormously.
Line chart with training progress on the horizontal axis and loss, described as average surprise, on the vertical axis. A single line starts high at about 9 and falls steeply at first, passing 4 early in training, then descends ever more slowly through 2.6, flattening toward about 1.9 by the end. The curve illustrates rapid early learning followed by slow, grinding improvement.
Training, in one plain sentence, is surprise minimization at industrial scale. Show the model a batch of text; compute the average loss; then adjust every one of the model’s Weights a tiny step in whichever direction would have made the true tokens slightly more probable — the recipe called gradient descent, which the Pre-training domain unpacks. Repeat, over trillions of tokens. Nobody edits rules into the model; ‘learning’ is only this: billions of parameters drifting toward whatever configuration makes real text less surprising.
Loss values like 2.0 are hard to feel, so the field uses a friendlier reading: Perplexity, defined as e^loss (the exponential of the average cross-entropy). Perplexity has a lovely plain meaning: a perplexity of k means the model is, on average, as uncertain as if it were choosing uniformly among k equally likely options. A loss of 2.0 is perplexity e² ≈ 7.4 — roughly ‘hesitating between 7 or 8 candidates’ at each step. A model that always nailed the next token exactly would have perplexity 1; random guessing over the whole Vocabulary would put perplexity at the vocabulary size. Watching perplexity fall from the tens of thousands toward single digits is watching a model learn a language.
Key terms: Next-token prediction, Logits, Loss, Cross-entropy, Perplexity
Cross-entropy and perplexity, precisely
Over a sequence of N tokens, the training loss is the average negative log-probability the model assigned to each true token given everything before it:
L = −(1/N) · Σ_t log p(x_t | x_1 … x_(t−1))
Each term is the surprise at position t; L is the mean surprise per token. The name comes from information theory: this is the cross-entropy between the true data distribution and the model’s predicted distribution, and the idea of measuring information as −log p goes back to Claude Shannon’s 1948 paper A Mathematical Theory of Communication, which founded the field. Shannon’s entropy is the surprise you would suffer using the true distribution itself — the floor. Cross-entropy is what your model suffers, and it can never beat that floor: the gap between the two is exactly how much your model’s beliefs diverge from reality. Training pushes toward the floor; it cannot tunnel below it, because language itself carries irreducible uncertainty — remember ‘My favorite color is ___’.
Perplexity is the exponential of the mean:
PPL = e^L
One unit convention to watch: with natural logarithms, loss is measured in nats and PPL = e^L; with base-2 logarithms it is measured in bits and PPL = 2^L. The perplexity comes out identical either way — only the logarithm base changes. The ‘choosing among k options’ reading is exact for uniform uncertainty: hesitating evenly among k candidates gives p = 1/k each, so L = log k and PPL = k. Real distributions are not uniform, so perplexity is best read as an effective branching factor. And a comparison caveat that matters in practice: perplexity is computed per token, so models with different tokenizers slice the same text differently and their perplexities are not directly comparable — the same trap as comparing per-token prices, wearing a lab coat.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.