Attention Arrives
Lesson 3 of 4 in From N-grams to Transformers.
By 2014, recurrent networks were good enough to attempt whole tasks end to end. Sequence-to-sequence learning (Seq2seq — Sutskever, Vinyals & Le, 2014, arXiv:1409.3215) hooked two LSTMs together: an encoder reads the input sentence and compresses it into its final hidden state; a decoder starts from that vector and generates the output, one token at a time. Machine translation with a single trained network — no hand-built pipeline of dictionaries and phrase tables. It worked, and it exposed its own flaw.
Everything the decoder will ever know about the source sentence must squeeze through that one fixed-size vector. Short sentences fit; long ones blur. Researchers watched translation quality sag as sentences grew — the fixed-vector bottleneck, the RNN’s forgetting problem promoted to a design flaw you could measure.
The fix (Bahdanau, Cho & Bengio, published at ICLR 2015, arXiv:1409.0473) was to stop compressing. Keep the encoder’s state at every input position, and each time the decoder writes a word, let it look back at all of them: compute a learned relevance score for each input position, Softmax the scores into weights, and take the weighted average. They called the mechanism Attention — the decoder attends to whichever input words matter for the word it is writing right now, with fresh weights for every output step. The bottleneck dissolved. And a lovely byproduct: the learned weights often lined up with genuine word alignments between the two languages — an interpretable map of what the model was consulting, hand-coded by no one.
Arc diagram over the sentence “The animal didn’t cross the street because it was tired”. A thick arc links “it” to “animal”, a thin arc links “it” to “street”, and a medium arc links “cross” to “animal”, illustrating that attention connects related words directly regardless of distance.
Attention entered the world as a patch on recurrence. The 2017 question was more radical: what if it is the whole engine? Attention Is All You Need (Vaswani et al., 2017, arXiv:1706.03762) deleted the RNN entirely. In the Transformer, every token computes attention over the other tokens directly — Self-attention — and a stack of such layers, alternating with small feed-forward networks, does all the work. No state marching left to right; word order enters through separate position signals instead.
Dropping recurrence buys two things at once. Direct long-range access: any token can reach any other in one step — no relay through hundreds of intermediate states, no fading trace to preserve. Parallel training: with no step-199-before-step-200 dependency, every position in a sequence — and every sequence in a batch — is processed simultaneously, as a few giant matrix multiplications. That is precisely the workload GPUs are built for. The price: attention compares every pair of tokens, so its cost grows with the square of sequence length. The field took that trade, and the next lesson is about what taking it made possible.
Path length, sequential steps, and the quadratic price
Two numbers make the recurrence-versus-attention trade precise — both from the comparison the transformer paper itself draws.
Maximum path length: how many computational steps separate two positions that need to interact. In a recurrent net the signal between positions i and j traverses every state in between — the path grows with their distance. In Self-attention the path is one step, regardless of distance. Long-range dependencies are easier to learn when the learning signal does not have to survive a long relay.
Sequential operations: a recurrent layer needs n strictly ordered steps to process n tokens; a self-attention layer needs a constant number, however long the sequence, because the whole thing is batched matrix multiplication. That is the property that let training runs saturate fleets of parallel accelerators — the practical unlock behind the next lesson.
The bill for both: each attention layer performs roughly n² pairwise comparisons where recurrence performed n updates. At 2017 sentence lengths that was cheap. Making it affordable at today’s Context window lengths is an engineering discipline of its own — the Inference & Serving domain covers it. And the full mechanism — queries, keys, values, Causal masking — is opened up term by term in Self-Attention.
Tool: Attention Visualizer — See the motif live: the Attention Visualizer traces which earlier tokens a model attends to as it processes each word of a sentence.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.