The Idea: Looking Things Up

Lesson 1 of 4 in Self-Attention: Every Token Looks at Every Token.

After the embedding and position steps, every Token in your prompt is a vector — one fixed point in space per token. That is not enough, because meaning depends on neighbours. In ‘the river bank’ and ‘the central bank’, the vector arriving for ‘bank’ is identical. Something has to let each token pull in information from the rest of the sentence.

That something is Attention. The idea predates the Transformer — it was invented for machine translation, so a decoder could glance back at the relevant source words (Bahdanau et al., 2014). The transformer’s move (Vaswani et al., 2017) was to make it the main mechanism: Self-attention, where a sequence attends to itself. Every token gets to ask a question of every token before it, and the answers flow in as a weighted blend.

Arc diagram over the sentence ‘The animal didn’t cross the street because it was too tired’. Thick arc from ‘it’ back to ‘animal’ (weight 0.7), thinner arcs from ‘it’ to ‘street’ (0.2) and from ‘tired’ to ‘animal’ (0.5).

What ‘it’ attends to. Arc thickness sketches attention weight: the token ‘it’ draws mostly on ‘animal’, a little on ‘street’. Illustrative weights on a classic example sentence — not real model output. (illustrative — source: Example sentence popularized by Vaswani et al. (2017))

The right mental model is a lookup — but soft, and learned.

A hash table or dictionary lookup is hard: one exact key match, one result, everything else ignored. Attention relaxes both properties. It matches by similarity, not exact equality, so every position gets some score. And instead of returning one winner, it returns a weighted average of what all positions offer — mostly ‘animal’, a little ‘street’, a trace of everything else. Nothing is ever fully ignored; relevance is a matter of degree.

Learned is the other half. Nobody wrote a rule that pronouns should look at nouns. The model arrived at this pattern during training because attending to the right noun made the next word easier to predict. What each token asks, and what it offers in return, come from learned Weights — parameters tuned by gradient descent — which is exactly where the next lesson goes.

Tool: Attention Visualizer — Watch attention weights form over a sentence you type — pick a token and see where it looks.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.