Why Many Heads Beat One
Lesson 1 of 3 in Multi-Head, Multi-Query, Grouped-Query Attention.
In the last module you built one head of Self-attention: every Token forms a query, scores every earlier token’s key, and takes a softmax-weighted mix of their values. Notice what that construction gives each position: exactly one attention distribution — one Softmax over the context, one set of Attention scores, one blended answer.
One distribution is a real limitation, because a token usually needs several different things from its context at the same time. A relationship that matters for grammar, a relationship that matters for meaning, and a relationship that matters for copying a name all pull the same distribution in different directions. A single head has to average those pulls, and averaged attention is blurry attention.
The fix in Vaswani et al.’s original Transformer paper is almost embarrassingly direct: don’t build one big head, build many small ones. That is Multi-head attention.
A vertical stack read bottom to top. Bottom: the input hidden state, one vector per token of width d_model. Next: per-head query, key, and value projections, h independent learned maps into small subspaces of width head_dim equals d_model divided by h. Middle, emphasized: h attention heads running in parallel, each computing its own scores and its own weighted mix of values. Next: the h head outputs concatenated back to width d_model. Top: the output projection W_O, which mixes what the heads found into one vector per token.
The arithmetic is the part people miss: many heads cost roughly what one wide head costs. Each head projects the token’s vector down into its own small subspace — Head dimension is typically d_model (model dimension) divided by the head count, so the original paper’s base model ran 8 heads of 64 dimensions over a 512-wide model. Total width in, total width out; the compute budget barely moves.
What you buy with that reshuffle is independence. Each head has its own learned query, key, and value projections, so each head measures its own notion of relevance and produces its own softmax distribution. One head can lock onto the subject of the sentence while another tracks the previous token and a third looks for an earlier mention of the current name — sharply, simultaneously, without competing for probability mass. The concatenation and output projection then write the combined findings back into the Residual stream for the next layer to use.
Key terms: Multi-head attention, Attention head, Head dimension, d_model (model dimension), Query, key, value (Q/K/V)
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.