Assembling the Block

Lesson 4 of 4 in Inside the Block: FFN, Residuals, Normalization.

You now hold all the parts. A modern decoder block is two sublayers, each wired with the same three-step pattern — normalize, transform, add:

x = x + Attention(Norm(x)) x = x + FFN(Norm(x))

That is the entire block. An LLM is this block repeated N times, with an embedding below, a final norm and Unembedding above, and causal masking inside every attention. What varies between model families is the seasoning, not the dish: LayerNorm or RMSNorm, plain or gated FFN, full multi-head attention or GQA to shrink the KV cache, and — in mixture-of-experts models — the dense FFN swapped for a routed bank of experts. The skeleton has barely changed since 2017; the block you can now draw from memory is, to a first approximation, every frontier model.

One modern pre-norm transformer block

  1. x — hidden states in
  2. Norm

    Rescale each token’s vector before attention (RMSNorm in most recent models).

  3. Multi-head attention

    Tokens exchange information under the causal mask. The only cross-position step.

  4. Add: x + attention output

    The attention edit is written back to the residual stream.

  5. Norm

    Rescale again before the FFN.

  6. Gated FFN (e.g. SwiGLU)

    Per-token expand-transform-squeeze. Holds roughly two-thirds of the block’s parameters.

  7. Add: x + FFN output

    The FFN edit joins the stream.

  8. On to block n+1

Interactive sorting exercise: Sort each component by the job it does inside the block.

Parameter accounting for one block

Write d for d_model and count weight matrices, ignoring biases and the (negligible) norm parameters.

Attention. Full multi-head attention projects the stream four times: W_Q, W_K, W_V, and the output projection W_O, each d × d once the heads are laid side by side. Total: 4d². (With MQA/GQA, W_K and W_V shrink because heads share keys and values, so real models often sit somewhat below 4d².)

FFN, classic recipe. Up-projection d × 4d plus down-projection 4d × d: 8d².

FFN, SwiGLU at parameter parity. Three matrices of d × d_ff with d_ff = (8/3)d: 3 · (8/3)d² = 8d² again — the two-thirds rule exists precisely to land on the same budget.

Norms. 2d (LayerNorm) or d (RMSNorm) per norm — millions of times smaller than the matrices. Ignore them.

Total per block ≈ 12d², of which the FFN is 8d² — two-thirds. Stack N blocks and the trunk of a dense model is ≈ 12 · N · d² parameters, plus the embedding/unembedding matrices at the ends.

Two consequences worth memorizing. First, per-token compute follows the same split: a matrix multiply costs about two FLOPs per weight per token, so forward compute per token is roughly 2 × the parameters touched (the rule of thumb used in scaling-law work, e.g. Kaplan et al., 2020, arXiv:2001.08361) — and two-thirds of that is FFN. Second, width is quadratic: double d and every block’s parameters and matmul FLOPs go up ~4×, not 2×. (Attention’s score computation adds a further term that grows with context length — that story gets its own module.)

Horizontal bar chart with two bars. Feed-forward network: 67 percent of block parameters. Attention projections: 33 percent. The FFN bar is twice the length of the attention bar.

Share of one dense block’s weight-matrix parameters, derived from the standard formulas: attention 4d² of 12d², FFN 8d² of 12d². Biases and norm parameters (thousands, not millions) are omitted; MQA/GQA shrink the attention share further. (calculated — source: Derived from Vaswani et al. (2017), arXiv:1706.03762 and Shazeer (2020), arXiv:2002.05202)

In production

The FFN is where a dense model spends most of its parameters and, per token, most of its floating-point operations — and those costs scale with the square of d_model. That one mechanism explains more of your serving bill than any other fact in this domain.

AWS

On GPU-backed inference — self-managed on EC2-class instances or behind SageMaker or Bedrock endpoints — the dense FFN matmuls are the compute-heavy inner loop that sets your tokens-per-second ceiling. Because per-token FLOPs grow roughly quadratically with model width, a modestly wider model can silently halve per-accelerator throughput, which surfaces as more accelerator-hours for the same traffic. Batching recovers efficiency precisely because FFN weights are read once and reused across every sequence in the batch.

Azure

Capacity planning for provisioned-throughput deployments (Azure OpenAI / AI Foundry) or AKS-hosted open models tracks per-token compute, which the FFN dominates. A wider model needs quadratically more compute per token, so the same throughput target costs proportionally more provisioned capacity. Memory follows the same law: roughly 12·d² weights per block times the layer count must fit in accelerator memory, or you shard across devices — and sharding adds interconnect hops that show up as latency.

Google Cloud

TPUs are built around systolic-array matrix units, and the FFN’s big dense matmuls are exactly the shape they like — utilization is usually highest in the FFN, on TPU or GPU alike. But the d² law is hardware-independent: doubling width quadruples per-block work on any chip Vertex AI can offer you. This is why sparse mixture-of-experts designs target the FFN specifically — more stored capacity without proportional per-token FLOPs, as the next module explains.

Key terms: Feed-forward network (FFN), SwiGLU, Residual stream, LayerNorm, RMSNorm, Pre-norm

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.