Positional Encoding: Teaching Order to a Set

Attention is order-blind by construction. Sinusoidal and learned positions, RoPE, and how position schemes shape context extension.

Content last verified 2026-09.

Lessons

1. Attention Is Order-Blind
2. Sinusoidal and Learned Positions
3. RoPE: Rotating Position into Attention
4. Stretching Positions: Context Extension

Sources

• Vaswani et al. (2017), Attention Is All You Need — arXiv:1706.03762
• Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — arXiv:2104.09864
• Chen et al. (2023), Extending Context Window of Large Language Models via Positional Interpolation — arXiv:2306.15595