The Feed-Forward Network
Lesson 1 of 4 in Inside the Block: FFN, Residuals, Normalization.
In the attention modules you watched tokens look at each other: multi-head attention is how information moves between positions. That is only half of a Transformer block. The other half never looks sideways at all.
The feed-forward network (FFN) takes each token’s vector — one hidden state of width d_model — and pushes it through a small two-layer neural network. The same Weights are applied to every position, and every position is processed independently. During the FFN, token 7 learns nothing from token 3. A useful (if loose) slogan: attention looks, the FFN thinks — first the block gathers context from other tokens, then it processes what each token now holds.
The FFN’s shape is an hourglass lying on its side: expand, transform, squeeze back. The first linear layer widens the vector from d_model to an inner size d_ff — classically 4× wider (the original transformer used 512 → 2048; Vaswani et al., 2017). A nonlinear activation function fires in that wide space, and a second linear layer projects back down to d_model so the result can rejoin the residual stream.
A vertical stack of layers forming one transformer block, read bottom to top: hidden states enter, pass a normalization layer, then multi-head attention where tokens exchange information, then a residual add. A second normalization follows, then the highlighted feed-forward network which expands each token vector about four times wider and squeezes it back, then a final residual add producing the output hidden states.
Modern models rarely use the original recipe unchanged. Most replace the plain two-matrix FFN with a gated variant. In SwiGLU (Shazeer, 2020, arXiv:2002.05202) the input is projected up twice in parallel: one projection passes through the Swish activation and acts as a gate, multiplying the other projection element by element before the down-projection. The gate lets the network dynamically dampen or pass each channel depending on the token — a small change that reliably improved language-model quality in Shazeer’s experiments. Gated FFNs of this family are standard in recent open models such as the Llama series.
Because SwiGLU needs three weight matrices instead of two, d_ff is commonly shrunk to about two-thirds of the classic 4× width so the Parameter count stays comparable — a detail that confuses many people reading model configs for the first time.
And here is the headline fact for the whole module: with either recipe, the FFN is where most of a dense transformer’s parameters live — roughly two-thirds of each block, as you will verify by hand in the last lesson. When you hear a model described by its parameter count, you are mostly counting FFN weights.
The FFN in equations
The original FFN (Vaswani et al., 2017) is two linear maps around a ReLU:
FFN(x) = W2 · max(0, W1 · x + b1) + b2
with W1 of shape d_model × d_ff, W2 of shape d_ff × d_model, and d_ff = 4 · d_model in the paper’s configurations. Applied to each position separately — equivalently, a 1×1 convolution over the sequence.
SwiGLU (Shazeer, 2020) replaces this with a gated form:
FFN(x) = W2 · ( Swish(W_gate · x) ⊙ (W_up · x) )
where ⊙ is elementwise multiplication and Swish(z) = z · sigmoid(z) (also called SiLU). Two up-projections (W_gate, W_up) plus one down-projection (W2) makes three matrices of size d_model × d_ff each, so parameter parity with the classic 2 · d_model · 4d_model layout is achieved by setting d_ff ≈ (2/3) · 4 · d_model = (8/3) · d_model — which is exactly the multiplier you will find in many open-model config files, rounded to a hardware-friendly value.
Why does gating help? Shazeer’s paper is refreshingly honest: it offers no explanation, attributing the success — ‘as all else, to divine benevolence’. The empirical result stood, and the field adopted it.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.