The Residual Stream

Lesson 2 of 4 in Inside the Block: FFN, Residuals, Normalization.

Look back at the block diagram and notice what happens around each sublayer: the input is added back to the output. Neither attention nor the FFN replaces the token’s vector — each contributes an edit: x + Sublayer(x).

This trick — residual (skip) connections — predates the transformer. He et al. (2015, arXiv:1512.03385) showed that very deep vision networks became trainable when each layer learned a change to its input rather than a whole new representation. The transformer adopted the same pattern around every sublayer, and it is a large part of why stacks of dozens of blocks train at all.

Interpretability researchers gave the resulting picture a name that has become standard vocabulary: the residual stream (the framing is developed in Anthropic’s transformer-circuits work — a lens on the architecture, not a separate mechanism). Instead of layers passing outputs to each other, imagine one vector per token — width d_model, nothing more — flowing unchanged from the Embedding at the bottom to the Unembedding at the top. Every sublayer is a tap on that pipe: it reads the current value, computes something, and writes its result back by addition.

The residual stream: one vector per token, many taps

  1. Embedding writes the stream

    The token’s embedding (plus any position information) is the stream’s initial value.

  2. Block 1 · attention tap

    Reads the stream at every position, moves information between tokens, writes its output back by addition.

  3. Block 1 · FFN tap

    Reads only this token’s vector, transforms it, writes the edit back by addition.

  4. Blocks 2 … N: same two taps, repeated

    Every block reads from and writes into the same d_model-wide vector. Later blocks see the accumulated edits of all earlier ones.

  5. Final norm

    In pre-norm models, one last rescaling before the stream is read out.

  6. Unembedding reads the stream

    The final value of the stream is projected against the vocabulary to produce logits.

Why does this framing earn its keep? Three consequences.

Gradients get a highway. Because the stream passes through each block via an identity path, the training signal can flow from the loss at the top to the earliest layers without being squeezed through dozens of transformations. That is the original point of residual learning, and it is what makes 30-, 60-, 100-block stacks optimizable.

The stream is a bandwidth budget. Everything any layer wants to tell any later layer must fit in the same d_model dimensions — there is no side channel. Attention heads, FFN outputs, position information: all of it is superposed in one vector. Interpretability work suggests different components read from and write to different subspaces of the stream, like radio stations sharing a band — a helpful picture to hold, even where the details are active research.

The default is “keep”. A block that has nothing useful to add can write (approximately) nothing, and the token’s representation survives untouched. Layers are not obligated to reconstruct everything — they accumulate refinements. This is why the same vector can carry a token’s identity from the embedding all the way to the unembedding at the top of the stack.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.