The Stack, End to End

Lesson 1 of 4 in One Token's Journey: The Full Forward Pass.

Over the previous modules you met every part of this machine one piece at a time: the Tokenizer that turns text into tokens, the Embedding matrix that turns ids into vectors, Positional encoding that teaches order to an order-blind mechanism, Self-attention and its many heads, and the Feed-forward network (FFN), Residual stream, and norms inside each block. This module bolts the pieces together and runs the whole machine once, end to end.

Assembled, a decoder-only Transformer is a surprisingly short list. Text goes in. A tokenizer converts it to token ids. An embedding lookup converts ids to vectors and position information enters. Then the same block — attention plus FFN, wrapped in residual connections and normalization — repeats N times. A final norm, a single Unembedding matrix multiply, and a Softmax turn the last vector into a probability for every entry in the Vocabulary. That is the entire forward pass of a modern Large language model (LLM).

A vertical stack diagram read bottom to top: input text enters a tokenizer producing token ids, an embedding layer with position information turns ids into vectors, N identical transformer blocks (attention plus feed-forward network, with residuals and norms) refine the vectors, then a final normalization, an unembedding layer producing one logit per vocabulary entry, and a softmax yielding the next-token probability distribution.

The full decoder-only transformer stack, read bottom to top. Every modern chat model is a variation on this assembly — the diagram is a generic teaching sketch, not any specific model. (illustrative — source: Vaswani et al. (2017), arXiv:1706.03762)

One property makes the whole stack legible: the shape never changes in the middle. From the embedding layer to the final norm, each token position is a vector of width d_model (model dimension), and every block reads from and writes back to that same vector — the Residual stream. Depth is not a pipeline of different machines; it is the same kind of machine applied N times, each pass refining the Hidden state a little further. Only at the two ends does the shape change: the Tokenizer maps text into ids, and the Unembedding maps the final vector back out to one score per Vocabulary entry.

The forward pass as a table — each stage, what it consumes, what it produces, and the module that taught it.
Stage	Consumes	Produces	Taught in
Tokenizer	Raw text	Token ids	Tokenization
Embedding + position	Token ids	One vector of width `d_model` per token	Embeddings, Positional Encoding
Transformer block × N	Token vectors	Refined token vectors — same shape	Self-Attention, Multi-Head, Inside the Block
Final norm	The residual stream	A normalized final hidden state	Inside the Block
Unembedding	The last position’s hidden state	Logits — one raw score per vocabulary entry	Embeddings
Softmax	Logits	The next-token probability distribution	This module

Key terms: Residual stream, Hidden state, d_model (model dimension), Unembedding, Feed-forward network (FFN), Attention

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.