What Lives in Embedding Space
Lesson 2 of 4 in Embeddings: Tokens Become Vectors.
Why would rows of learned numbers mean anything? Nobody told the model that “cat” and “kitten” are related. The answer is training pressure. To win at Next-token prediction, the model must treat tokens that show up in similar contexts similarly — whatever helps it predict the words around “cat” mostly helps around “kitten” too. The cheapest way for the network to exploit that is to give such tokens nearby vectors, so all downstream machinery can process them alike.
The result is the linguist J.R. Firth’s old slogan — “you shall know a word by the company it keeps” — implemented in geometry. Distance and direction in embedding space reflect statistical behaviour in the training data, which usually tracks meaning: synonyms cluster, related concepts sit near each other, unrelated ones end up far apart.
Horizontal bar chart of five word pairs with cosine similarity values: cat and kitten 0.92, cat and dog 0.81, Paris and France 0.65, cat and car 0.18, cat and democracy 0.07.
Two habits of thought will keep you honest here. First, similarity means “behaves alike in text,” not “is a dictionary synonym.” “Cat” and “dog” score high because they keep the same company, not because they mean the same thing. Second, spelling is invisible: the vectors are attached to token ids, so surface resemblance between strings counts for nothing.
Directions in the space can carry structure too — differences between vectors sometimes encode relations like singular→plural or country→capital. That observation launched a famous claim, which deserves its own warning label.
Measuring closeness
The standard similarity measure is cosine similarity: cos(a, b) = (a · b) / (‖a‖ ‖b‖) — the dot product of two vectors divided by the product of their lengths. It ranges from −1 (opposite directions) through 0 (perpendicular) to 1 (same direction), and it ignores vector length, comparing only direction. The unnormalized cousin, the raw dot product a · b, reappears at the heart of Attention in a later module — the transformer is full of “how aligned are these two vectors?” questions.
One high-dimensional surprise worth knowing: in spaces with thousands of dimensions, two random vectors are almost always nearly perpendicular. So even modest positive cosine values can signal a real relationship, and similarity scores from an embedding model tend to bunch inside a narrow band rather than spreading across the full −1 to 1 range. Compare scores against each other, not against an absolute scale.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.