LayerNorm, RMSNorm, and Where They Sit

Lesson 3 of 4 in Inside the Block: FFN, Residuals, Normalization.

The residual stream has a failure mode: it is a running sum. Dozens of blocks each add their edit, and nothing so far guarantees the total stays at a sensible magnitude. Vectors that drift too large or too small make training unstable — gradients explode or vanish, and the network chases a moving target.

Normalization is the thermostat. Before (or after) each sublayer, the token’s vector is rescaled to a standard magnitude so that every sublayer sees inputs in the range it was tuned for. Two recipes dominate, and they differ by one honest question: how much of the classic statistics do you actually need?

LayerNorm

LayerNorm (Ba, Kiros & Hinton, 2016, arXiv:1607.06450) works per token, across the d_model features:

  1. subtract the vector’s mean (re-center),
  2. divide by its standard deviation (re-scale),
  3. apply a learned per-feature gain γ and bias β.

Two learned vectors → 2 · d_model parameters per norm. Crucially, unlike batch normalization it never looks at other examples or other positions — every token normalizes itself, which is exactly what variable-length sequence models need. This is the norm of the original transformer and the early LLM lineage.

RMSNorm

RMSNorm (Zhang & Sennrich, 2019, arXiv:1910.07467) asks: was the re-centering ever doing anything? Their answer: mostly no. RMSNorm

  1. skips the mean entirely,
  2. divides by the root mean square of the features,
  3. applies the learned gain γ only — no bias.

Half the statistics, half the learned parameters (d_model), fewer operations — and in their experiments, quality on par with LayerNorm. That trade won: RMSNorm is the default in most recent open models, including the Llama family.

Which norm you pick matters less than where you put it — and this is where a famous diagram misleads. The 2017 paper placed the norm after the residual add: Norm(x + Sublayer(x)) — post-norm. The norm sits on the stream itself, re-scaling the accumulated sum at every block.

Modern models almost universally move it inside the branch: x + Sublayer(Norm(x)) — Pre-norm. The sublayer gets a cleanly normalized input, but the stream itself flows through the block untouched except for the addition. The identity highway from the last lesson stays genuinely identity.

The difference shows up in training stability. Analysis of the two layouts (Xiong et al., 2020, arXiv:2002.04745) showed that in post-norm transformers, gradients near the output are large at initialization and the architecture leans on careful learning-rate warmup to avoid diverging; pre-norm keeps gradients well-behaved as depth grows, and trains reliably where deep post-norm stacks are fragile. GPT-2 had already moved the norm to the front, and pre-norm has been the default recipe for large models since. One bookkeeping consequence: because no norm sits on the stream any more, pre-norm models add a single final norm after the last block, just before the unembedding.

Post-norm vs pre-norm: same ingredients, different wiring — and very different training behavior.
PlacementPost-norm (2017 original)Pre-norm (modern default)

Formula

Norm(x + Sublayer(x))

x + Sublayer(Norm(x))

Where the norm sits

On the stream — every block rescales the accumulated sum

On the branch — the stream passes through unchanged except for additive edits

Identity path

Interrupted at every block

Clean from embedding to final norm

Training at depth

Fragile — typically needs careful learning-rate warmup (Xiong et al., 2020)

Stable — the standard choice for deep LLM stacks

Extra piece

None

One final norm after the last block, before the unembedding

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.