RoPE: Rotating Position into Attention

Lesson 3 of 4 in Positional Encoding: Teaching Order to a Set.

Additive schemes stamp position onto a token once, at the bottom of the stack, and hope the signal survives dozens of layers. But think about what attention actually needs. When a verb looks for its subject, what matters is rarely ‘the subject is at absolute position 847’ — it is ‘the subject is a few tokens back’. Relative offsets are the currency of attention; absolute stamps are a roundabout way to pay in it.

Rotary position embedding (RoPE), introduced by Su et al. (2021), injects position exactly where the comparison happens. Instead of adding anything to the embeddings, RoPE takes each query and key vector inside the attention layer and rotates it — by an angle proportional to the token’s position. Position 5 gets five clicks of rotation, position 100 gets a hundred. Each 2-D slice of the vector spins at its own frequency, reusing the fast-to-slow spectrum you met in the sinusoidal lesson.

The payoff is a small piece of geometry: when you take the dot product of a query rotated by m clicks and a key rotated by n clicks, the two absolute rotations cancel into one relative rotation of n − m clicks. The attention score ends up depending on the two tokens’ content and their offset — and nothing else. ‘Three tokens back’ produces the same geometric relationship at position 10 as at position 10,000.

How RoPE turns absolute positions into relative offsets

  1. Token vectors

    Hidden states enter the attention layer with no position added — RoPE does not touch the embeddings.

  2. Project to queries and keys

    The usual learned projections produce q for the token at position m and k for the token at position n.

  3. Split each vector into 2-D pairs

    A head of width d becomes d/2 little 2-D planes, each with its own rotation frequency θᵢ.

  4. Rotate: q by m·θᵢ, k by n·θᵢ

    Each pair is rotated by an angle proportional to its token’s absolute position. Values are left unrotated.

  5. Dot product q · k

    Rotations compose: the m-rotation and n-rotation collapse into a single rotation by (n − m)·θᵢ.

  6. Score depends on content + offset (n − m) only

    Absolute positions cancel. The same relative pattern — ‘attend three back’ — costs the same everywhere in the sequence.

Three practical consequences made RoPE the default choice in most recent open-weight architectures (the Llama, Mistral, and Qwen families all document it, among many others):

  • It lives in every layer. The rotation is applied inside each attention computation, at every layer and every head — position information cannot fade on the way up the stack the way a bottom-of-stack additive stamp can.
  • It costs nothing to store. The angles come from a formula; there is no position table, no extra parameters.
  • It cooperates with the KV cache. Keys are cached already-rotated by their absolute position; because scores only ever depend on offsets, a new query compares correctly against every cached key with no recomputation.
The rotation math, worked through

Work in one attention head of width d (the Head dimension). RoPE partitions the query vector into d/2 two-dimensional pairs and assigns pair i a frequency

θᵢ = 10000^(−2i/d), for i = 0, 1, …, d/2 − 1

— the same geometric spectrum as the sinusoidal scheme: pair 0 rotates fast, the last pair barely moves. For a token at position m, each pair (x, y) of the query is multiplied by the 2-D rotation matrix

R(mθᵢ) = [[cos mθᵢ, −sin mθᵢ], [sin mθᵢ, cos mθᵢ]]

Keys get the identical treatment with their own position n. Values are never rotated — position should shape who attends to whom, not what gets copied.

Now the cancellation. Rotation matrices compose by adding angles, and the transpose of a rotation is its inverse: R(a)ᵀ = R(−a) and R(−a)·R(b) = R(b − a). So for one pair, the attention score contribution is

(R(mθ)q)ᵀ · (R(nθ)k) = qᵀ · R(mθ)ᵀ R(nθ) · k = qᵀ · R((n − m)θ) · k

The absolute positions m and n have vanished individually; only their difference survives. Summing over all d/2 pairs gives the full dot product, and the argument holds pair by pair.

The complex-number view is even tidier. Treat pair i as a complex number z. RoPE maps q → q·e^(imθᵢ) and k → k·e^(inθᵢ), and the real inner product becomes Re(q̄k · e^(i(n−m)θᵢ)) — a function of content and offset alone. Su et al. also show that with this frequency spectrum, the score contribution tends to decay as the offset grows, a useful inductive bias: nearby tokens interfere constructively more easily than distant ones.

One more detail worth having straight: RoPE still knows absolute position — each cached key physically carries its own rotation. What is relative is the score. That distinction is exactly what the next lesson exploits when models stretch their positions.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.