Embeddings: Tokens Become Vectors

The embedding matrix, what geometry in representation space means, contextual versus static embeddings, and the unembedding back to logits.

Content last verified 2026-09.

Lessons

1. From Token IDs to Vectors
2. What Lives in Embedding Space
3. Contextual vs Static Embeddings
4. The Output Side: Unembedding and Logits

Sources

• Vaswani et al. (2017), “Attention Is All You Need” — arXiv:1706.03762
• Mikolov et al. (2013), “Efficient Estimation of Word Representations in Vector Space” — arXiv:1301.3781
• Press & Wolf (2016), “Using the Output Embedding to Improve Language Models” — arXiv:1608.05859
• Peters et al. (2018), “Deep Contextualized Word Representations” — arXiv:1802.05365