Attention Is Order-Blind

Lesson 1 of 4 in Positional Encoding: Teaching Order to a Set.

Here is an uncomfortable fact about the Transformer: the mechanism at its heart has no idea what order your words came in.

Attention works by comparing vectors. Each Token asks a question (its query), every token offers an answer (its key), and the Attention scores that decide who listens to whom are just dot products between those vectors — content compared against content. Position appears nowhere in that computation. If the input Embedding vectors carry no information about where a token sits, then the attention layer receives ‘dog bites man’ and ‘man bites dog’ as exactly the same bag of three vectors.

That is not a small technicality. Word order is most of syntax. A model that cannot see order cannot tell subject from object, negation from affirmation, or a question from its answer.

An arc diagram over the three tokens dog, bites, man. Arcs connect dog to bites with weight 0.55, man to bites with weight 0.55, and dog to man with weight 0.30. The caption explains that reordering the tokens would carry the same arcs along with them, because the weights depend only on token content.

Attention weights are computed from token content, not token position. Scramble the row — ‘man bites dog’ — and every arc follows its tokens to their new seats, unchanged. Weights here are toy values chosen to make the point (illustrative — not real model output). (illustrative — source: Vaswani et al. (2017), Attention Is All You Need — arXiv:1706.03762)

The precise name for this property is permutation equivariance: shuffle the input tokens and the outputs shuffle in exactly the same way, with every output vector computed identically. Attention treats its input as a set, not a sequence. (Contrast this with the recurrent networks that came before: they read left to right, so order was built in — along with a painful serial bottleneck. The transformer bought parallelism by giving order up.)

One nuance worth knowing: in decoder-only models, Causal masking breaks perfect symmetry — a token can only attend backwards, and researchers have shown such models can learn some positional signal even with no explicit encoding (Haviv et al., 2022, arXiv:2203.16634). But every major production architecture still injects position deliberately, because implicit order signals are weak and hard to control.

So the fix has to come from outside the attention equation: a Positional encoding stamps each token’s vector with where it sits — either by adding a position vector to its embedding before attention ever runs, or by weaving position into the query–key comparison itself. Those two roads are the next two lessons.

Interactive sorting exercise: A transformer with NO positional information reads each pair of inputs below. Sort each pair: can the model tell the two apart, or do they look identical to it?

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.