From Token IDs to Vectors

Lesson 1 of 4 in Embeddings: Tokens Become Vectors.

The Tokenizer left you holding a list of integers. That’s a problem, because a neural network can’t do anything meaningful with the number 8,943 as a number — token ids are arbitrary catalogue positions, like jersey numbers. Player 23 isn’t “one more” than player 22, and token 8,944 has no relationship to token 8,943 just because the ids are adjacent.

So the first thing a Transformer does is trade every id for something it can compute with: a vector — a list of numbers, typically a few hundred to a few thousand of them. The swap happens in the Embedding matrix: a single giant table with one row per entry in the Vocabulary. Token id 8,943 in? The model copies out row 8,943. That copy is the token’s Embedding, and it’s the form in which the token travels through everything that follows.

The whole operation is a lookup. No arithmetic, no cleverness — the intelligence is in what the rows contain, not in how they’re fetched.

Heatmap with six rows labeled cat, kitten, dog, car, the, and run, and eight columns labeled d1 through d8. Each cell shows an intensity between 0 and 1. The rows for cat and kitten show nearly identical patterns, dog is close to them, while car shows an almost inverted pattern, and the and run sit in the middle.

A toy embedding matrix: one row per vocabulary token, one column per dimension. Real models have tens of thousands of rows and thousands of columns; these six rows and eight dimensions use made-up values to show the shape of the thing. Notice that “cat” and “kitten” have similar row patterns while “car” does not — that similarity is learned, not typed in. (illustrative — source: Concept after Vaswani et al. (2017), arXiv:1706.03762)

The width of each row — how many numbers represent one token — is the model’s d_model (model dimension), and it’s one of the most consequential numbers in the whole architecture. Every vector flowing through the model keeps this width, layer after layer: it’s the width of the pipe everything else is built around.

Where do the row values come from? They’re learned parameters. At the start of training every cell is small random noise; every gradient step of Next-token prediction training nudges them, and after trillions of tokens the rows have organized themselves so that the rest of the network can predict well. Nobody assigns dimension 7 a meaning like “furriness” — whatever a dimension encodes is an accident of training, and meaning is usually smeared across many dimensions at once.

One more property to file away for later: the lookup is static. Token 8,943 fetches the same row whether it appears in a physics paper or a grocery list. Context enters the story only after this layer — that’s lesson three.

Key terms: Embedding matrix, d_model (model dimension), Hidden state, Unembedding, Residual stream

The lookup, precisely

Formally, the embedding matrix W_E has shape |V| × d_model, where |V| is the vocabulary size. If you write token id i as a one-hot row vector x (all zeros, a single 1 in position i), then the embedding is the matrix product x · W_E — which is exactly row i of W_E. “Lookup” and “multiply by a one-hot vector” are the same operation; implementations use the lookup because multiplying by a vector of zeros is wasted work.

The parameter bill is |V| × d_model. For a concrete open example: GPT-2’s byte-level BPE vocabulary has 50,257 entries and its smallest variant uses d_model = 768, so the embedding matrix alone holds 50,257 × 768 ≈ 38.6 million numbers — roughly a third of that model’s parameters. In today’s much larger models the embedding’s share shrinks, because block parameters grow roughly with d_model² per layer while the embedding grows only linearly with d_model.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.