A Probability for Every Token
Lesson 2 of 4 in Next-Token Prediction, Really.
When you played the game, you produced one guess, maybe two. A model does something stranger and more thorough: at every step, it scores every single entry in its Vocabulary — every word-piece, punctuation mark, and symbol its Tokenizer knows, typically tens of thousands to hundreds of thousands of them. Not just plausible candidates. All of them, every time.
The raw scores are called Logits — one number per vocabulary entry, higher meaning more favored. A function called Softmax then converts those scores into proper probabilities: all positive, summing to exactly 1. The result is a probability distribution over the whole vocabulary: a complete, ranked opinion about what could come next, from ‘ Paris’ at the top to some fragment of Hungarian at the bottom with a probability so small it needs scientific notation.
This is the single most useful mental picture in the whole field, so let it replace the folk image now. The model never outputs a word. It outputs a distribution. The word you see is chosen from that distribution afterwards, by a separate, much dumber piece of code.
Horizontal bar chart of an illustrative probability distribution over a vocabulary for one prompt. “ Paris” dominates at 87 percent, followed by “ the” at 4 percent, “ located” at 2 percent, “ a” at 1.5 percent, and “ one” at 1 percent. A final bar labeled “every other token combined” shows 4.5 percent, representing the long tail of tens of thousands of tiny but nonzero probabilities.
The long tail is not decoration — it is where a lot of observed behavior lives. Because every token keeps a nonzero score, unlikely continuations are merely unlikely, never impossible. Ask the same model the same question twice and you may get different answers; that is not a bug in the model but a choice at the last step. In one line: take the top token every time and output is deterministic; draw from the distribution — Sampling — and it varies. That one dial (and its refinements: temperature, top-p, and friends) belongs to the Inference & Serving domain, which treats it properly. Here, just keep the division of labor straight: the model produces the distribution, and a sampling policy picks from it.
How the distribution gets computed — embeddings, Attention, the whole Transformer stack — is the story of an entire domain. When you are ready for the mechanics, One Token’s Journey walks a single prediction end to end. For this module, the interface is what matters: text in, distribution over the vocabulary out, once per token.
Softmax: from scores to probabilities
The model’s final layer produces one logit per vocabulary entry: a vector z of length |V|, where |V| is the vocabulary size. Logits are unnormalized — they can be negative, and they do not sum to anything meaningful. Softmax fixes that:
P(token i) = e^(z_i) / Σ_j e^(z_j)
Read it in three moves. Exponentiate each logit: e^z is always positive, so no token can have negative probability, and because the exponential grows so fast, modest logit gaps become large probability ratios — a logit lead of 2 is already an e² ≈ 7.4× probability advantage. Sum the exponentials over the whole vocabulary — that is the Σ_j in the denominator, one term per vocabulary entry. Divide each token’s exponential by that sum, so everything lands strictly between 0 and 1 and totals exactly 1.
Two properties do a lot of quiet work downstream. First, softmax never outputs a hard zero — e^z is positive for any finite logit — which is precisely why every token stays reachable and sampled outputs can surprise you. Second, only logit differences matter: add the same constant to every logit and the distribution is unchanged. And one hook for later: dividing all logits by a constant T before softmax sharpens (T < 1) or flattens (T > 1) the distribution without retraining anything — that constant is the ‘temperature’ dial you will meet again in Inference & Serving.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.