Where the Parameters Live

Lesson 4 of 4 in One Token's Journey: The Full Forward Pass.

Say “seven billion parameters” and you have said almost nothing about where they are. In a dense decoder-only transformer, the learned Weights live in exactly four places: the Embedding matrix (one d_model (model dimension)-wide row per Vocabulary entry), the attention projections in every block (the Q, K, V and output matrices), the Feed-forward network (FFN) in every block (two big matrices connecting the stream to a wider hidden layer and back), and the Unembedding matrix (another vocabulary-sized matrix — unless it is tied to share weights with the embedding). Norm gains and biases exist too, but they are vectors, not matrices: a rounding error in the budget.

Two of the four scale with the vocabulary; the other two scale with depth × width². That split decides how the budget shifts as models grow — and, as the deep dive shows, it compresses into one famous formula.

A horizontal bar chart of parameter share: FFN weights across all blocks at about 63%, attention projections across all blocks at about 31%, the embedding matrix at about 3%, and the unembedding matrix at about 3%.

Parameter share by component for a generic dense decoder, derived from the standard counting formulas with illustrative dimensions: d_model = 4096, 32 layers, ~50k vocabulary, d_ff = 4·d_model, untied embeddings (≈6.9B total). A teaching configuration, not any specific model. (illustrative — source: Counting per Kaplan et al. (2020), arXiv:2001.08361)
The standard parameter count, derived

Write d for d_model, V for vocabulary size, and n_layer for the number of blocks. Then, for the classic dense architecture:

  • Embedding: V · d — one row per token.
  • Attention, per block: four d × d matrices W_Q, W_K, W_V, W_O → 4d².
  • FFN, per block: up and down projections d · d_ff + d_ff · d; with the classic choice d_ff = 4d, that is 8d².
  • Per block: 4d² + 8d² = 12d². Whole stack: N ≈ 12 · n_layer · d² non-embedding parameters — the order-of-magnitude rule the scaling-law literature runs on (Kaplan et al., 2020).

Worked on the figure’s toy configuration: d = 4096, n_layer = 32 gives 12 × 32 × 4096² ≈ 6.4B non-embedding parameters; a ~50k vocabulary adds roughly 0.2B on each side (embedding and unembedding).

Where the formula bends in modern architectures:

  • GQA / MQA shrink W_K and W_V by sharing key/value heads, so attention drops below 4d² per block.
  • SwiGLU FFNs use three matrices, 3 · d · d_ff, typically with d_ff ≈ (8/3)·d so the budget stays near 8d².
  • MoE multiplies the FFN term by the expert count while each token activates only a few experts — parameters stored far exceed parameters used per token (Sparse activation).
  • Tied embeddings reuse one V · d matrix for both embedding and unembedding.

One compute link worth memorizing: a dense forward pass costs roughly 2 FLOPs per parameter per token (≈ 2N, the standard approximation in Kaplan et al., 2020). Parameters are not just memory — they are your per-token compute bill.

The consequences read straight off the chart. In a dense model at this scale, the FFN is the biggest budget line — roughly two-thirds of every block under the classic d_ff = 4d — which is exactly why Mixture of experts (MoE) designs make the FFN sparse rather than attention. The vocabulary terms barely register here, but shrink the model and they roar back: in a small Dense model with a large multilingual vocabulary, embedding plus unembedding can claim a substantial fraction of all weights. And because a dense forward pass touches every Parameter once per token, where the parameters live is also where your compute goes.

Interactive sorting exercise: Sort each weight group by how its parameter count scales.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.