Self-Attention: Every Token Looks at Every Token

Queries, keys, and values; the scaled dot-product equation term by term; causal masking — the mechanism that made the transformer.

Content last verified 2026-09.

Lessons

1. The Idea: Looking Things Up
2. Queries, Keys, Values
3. The Attention Equation, Term by Term
4. Causal Masking: No Peeking at the Future

Sources

• Vaswani et al. (2017), Attention Is All You Need
• Bahdanau, Cho & Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate
• Jay Alammar, The Illustrated Transformer