From Logits to the Next Word
Lesson 3 of 4 in One Token's Journey: The Full Forward Pass.
The journey ended with one vector: the last position’s hidden state, freshly normalized. The Unembedding matrix — shape Vocabulary size × d_model (model dimension) — multiplies it, and out come Logits: one raw score for every vocabulary entry, typically tens of thousands of numbers at once.
Logits are not probabilities. They can be negative, they do not sum to anything in particular, and only their differences carry meaning — a logit of 7.1 beating a 6.7 expresses the same preference whether the pair is (7.1, 6.7) or (107.1, 106.7). Think of a logit as accumulated evidence: everything N blocks of attention and FFNs concluded, compressed into “how strongly does this vector point at this token?”
A horizontal bar chart of next-token probabilities: “ mat” at 32%, “ floor” at 21%, “ couch” at 12%, “ chair” at 9%, “ roof” at 5%, and all other vocabulary tokens combined at 21%.
Softmax turns scores into that distribution in two moves: exponentiate every logit, then divide each result by the sum of all of them. Exponentiation is why gaps matter so much — a modest logit lead becomes a large probability lead, and tokens a few logits behind the leader end up sharing crumbs.
One knob you will meet everywhere gets a preview here: temperature. Divide all logits by a constant T before the softmax. T < 1 widens the gaps, sharpening the distribution toward the favorite; T > 1 shrinks them, flattening the distribution so unlikely tokens get real probability mass; as T → 0 you approach greedy argmax. Temperature never changes the model’s opinion — the logits — only how boldly the choice is made from them. Everything deeper about Sampling (top-k, top-p, and friends) lives in the Inference & Serving domain.
In production
The forward pass is the unit of everything downstream: one full pass through all N blocks per generated token. That single mechanical fact shapes how every cloud meters, sizes, and streams LLMs.
AWS
Managed model APIs such as Amazon Bedrock meter input and output tokens separately, and output tokens typically carry the higher rate. The mechanism is this lesson: prompt tokens are processed together in one parallel pass over the sequence, while every generated token demands its own full forward pass through the stack — sequential work that cannot be parallelized within your request.
Azure
On Azure OpenAI in Foundry, latency budgets split along the same seam: time-to-first-token is the parallel pass over your prompt, and time-per-output-token is one sequential forward pass each. Long outputs, not long prompts, dominate wall-clock time — trimming a verbose prompt mostly saves token cost, while capping requested output length saves time.
Google Cloud
Streaming on Vertex AI exists because of the journey you just traced: each output token becomes available the moment its forward pass finishes, so the API can emit tokens as they are produced. Perceived latency then hinges on time-to-first-token, while total cost still scales with every sequential pass running behind the stream.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.