The Guess-the-Next-Token Game

Lesson 1 of 4 in Next-Token Prediction, Really.

Finish this sentence: The capital of France is ___.

You did not deliberate. Somewhere in your head, ‘Paris’ arrived before you decided to think about it. Now try these: Once upon a ___. 2 + 2 = ___. The opposite of hot is ___. Each one pulls your mind toward one continuation, hard.

Now try: My favorite color is ___. Still constrained — it should be a color — but no single color is forced. And The best movie ever made is ___ is barely constrained at all: grammar demands a noun phrase, and after that, taste takes over.

You have just experienced the entire objective an Large language model (LLM) is trained on. Given some text, guess what comes next. That is the whole game — not answering questions, not being helpful, not reasoning. Those come later, and this module’s last lesson explains how they grow out of the game. The model plays it one Token at a time: read everything so far, guess the next piece, append it, and guess again. A thousand-word essay is a thousand-odd rounds of the same move.

Horizontal bar chart of illustrative next-token probabilities for the prompt “The capital of France is”. The token “ Paris” has by far the largest bar at 87 percent. Smaller bars follow: “ the” at 4 percent, “ located” at 2 percent, “ a” at 1.5 percent, “ one” at 1 percent, and “ known” at 0.5 percent, showing one dominant continuation with plausible alternatives trailing behind.

Plausible continuations of ‘The capital of France is’, as a model might score them. One candidate dominates because grammar and geography both point at it — but notice that the alternatives are not nonsense, just less likely. Values are invented for teaching; real models assign real numbers like these to every token in their vocabulary. (illustrative — source: Shannon (1951) ran this game with people — Prediction and Entropy of Printed English)

Why could you play so well without trying? Because text is not random. Three kinds of signal make continuations predictable, and you used all three at once.

Grammar narrows the field. After The capital of France is, English all but demands a noun phrase. Whole categories of tokens — verbs, punctuation, conjunctions — are nearly ruled out before meaning even enters.

Facts narrow it further. Among noun phrases, one is overwhelmingly what writers of English actually put there. You have read (or heard) some version of this sentence many times; so has a model trained on a large slice of the written internet. Facts, in this game, are just extremely reliable patterns in text.

Context does the rest. She ordered a cup of ___ wants ‘coffee’ or ‘tea’ — but if the previous paragraph was set in a ramen shop, your guess shifts. Whatever came before changes what comes next, and a model reads everything in its Context window before every single guess.

Hold on to the other half of the lesson too: some continuations are genuinely open. When many next tokens are plausible, prediction cannot be certain — no matter how good the predictor gets. That is not a flaw; it is a property of language. It is also, as you will see next lesson, exactly why the model’s output has to be a set of scored options rather than one answer.

Interactive sorting exercise: Play the game deliberately: how constrained is the next token in each prompt? Drag each one to the bucket that fits.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.