Multi-Head, Multi-Query, Grouped-Query Attention

Why many small heads beat one big one, what heads actually learn, and how MQA/GQA trade attention quality for serving memory.

Content last verified 2026-09.

Lessons

1. Why Many Heads Beat One
2. What Heads Actually Learn
3. MQA and GQA: Sharing Keys and Values

Sources

• Vaswani et al. 2017 — Attention Is All You Need (arXiv:1706.03762)
• Olsson et al. 2022 — In-context Learning and Induction Heads (Anthropic)
• Shazeer 2019 — Fast Transformer Decoding: One Write-Head is All You Need (arXiv:1911.02150)
• Ainslie et al. 2023 — GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv:2305.13245)
• Voita et al. 2019 — Analyzing Multi-Head Self-Attention (arXiv:1905.09418)