Sinusoidal and Learned Positions
Lesson 2 of 4 in Positional Encoding: Teaching Order to a Set.
The first fix is the simplest one imaginable: build a vector that encodes ‘position 7’, and add it to the token embedding before the first attention layer runs. Every token now carries a superposition of two signals — what I am plus where I sit — and attention can learn to read either.
The original transformer paper (Vaswani et al., 2017) proposed a fixed, formula-based version: sinusoidal encodings. No learning involved — each position gets a deterministic pattern of sine and cosine values, one pair of dimensions at a time, each pair oscillating at a different wavelength.
A line chart with position 0 to 24 on the x-axis and encoding value from -1 to 1 on the y-axis. Three curves: dimension 0, sin(pos/1), oscillates rapidly through about four full cycles; dimension 16, sin(pos/10), rises smoothly to 1.0 near position 16 then begins to fall; dimension 32, sin(pos/100), rises almost linearly and only reaches about 0.24 by position 24.
Why sines and cosines rather than, say, just the number pos? Two properties. First, every value stays in [-1, 1] regardless of position, so position never drowns out the token embedding it is added to. Second — and this is the elegant part — for any fixed offset k, the encoding of pos + k is a fixed linear transformation (a rotation of each sine–cosine pair) of the encoding of pos. The paper chose the form precisely so that attending to ‘the token three back’ could be learned as a simple, position-independent operation. Hold that thought: the next lesson takes it to its logical conclusion.
The obvious alternative is to stop being clever and just learn the position vectors: allocate an embedding table with one row per position, look up row 7 for position 7, and train those rows by gradient descent like every other weight. Early encoder and decoder models such as BERT and GPT-2 took exactly this route, and it works well within its limits.
The limits are real, though. A learned table has nothing at all for position 1025 if it was built with 1024 rows — the model’s Context window is hard-capped by the table. And each position’s vector is learned independently, so the model gets no built-in notion that position 500 relates to 503 the way 100 relates to 103.
| Sinusoidal (fixed) | Learned absolute | |
|---|---|---|
How position is produced | Deterministic sine/cosine formula — no training | Trainable embedding table, one row per position |
Where it enters the model | Added to token embeddings before the first layer | Added to token embeddings before the first layer |
Extra parameters | None | max positions × d_model |
Beyond the trained length | Formula produces values for any position — but the model has never trained on them, so quality still degrades | No row exists — hard cap at the table size |
Relative offsets | Offset k is a fixed linear map of the encoding — learnable but indirect | No built-in structure; each position learned independently |
Key terms: Positional encoding, Sinusoidal encoding, Rotary position embedding (RoPE), Context extension, d_model (model dimension)
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.