Why Models Read Tokens, Not Words

Lesson 1 of 4 in Tokenization: Text Becomes Tokens.

An Large language model (LLM) never sees the text you send it. Before anything reaches the model, a Tokenizer chops your string into pieces called tokens and replaces each piece with an integer ID from a fixed list — the Vocabulary. The model consumes those integers, and everything it produces is chosen from that same list, one token at a time. Tokenization is the door to the whole machine, and every design choice behind that door leaks into behavior you will eventually have to debug.

So what should a piece be? There are three obvious candidates, and each fails in an instructive way.

Whole words feel natural and fail first. Language is open-ended: names, typos, code identifiers, and new slang mean no word list is ever complete. A word-level model needs an unknown token for everything outside its list — and every rare word collapses into the same meaningless blank. The vocabulary also balloons: every inflection (run, runs, running) needs its own entry and its own learned vector.

Single characters never hit an unknown word, but they stretch every sentence into a long crawl — each character becomes one position the model must process and, on the output side, one generation step. Meaning also lives painfully far from the input: the model has to rediscover that c-a-t is an animal from raw letters, spending layers on what a better unit would give it for free.

Raw bytes are the most universal alphabet of all — any text in any language is just bytes — but they make sequences even longer than characters do.

Bar chart showing the sentence "Transformers read subword tokens" split into six tokens: "Transform", "ers", " read", " sub", "word", and " tokens". Each bar is one token and its length shows the number of characters in that token, illustrating that frequent words stay whole while rarer words split into pieces.

One sentence, split into subword tokens. Common words survive whole; the rarer word splits into reusable pieces, and the space before a word travels with its token. Toy example — real tokenizers differ in exactly where they cut. (illustrative — source: Try real splits in the OpenAI interactive tokenizer)

Here is the part that matters even if you never open a model: the token is the meter of the entire LLM economy.

  • The Context window — how much the model can consider at once — is a budget denominated in tokens, not words or pages.
  • API pricing is per token, input and output metered separately. Your prompt template, your retrieved documents, your few-shot examples: all of it is token count before it is anything else.
  • Generation latency scales with output tokens, because a transformer produces them one prediction at a time.

Two consequences follow. First, the same text is a different number of tokens under different tokenizers — so cost comparisons between models that quote per-token prices are meaningless until you tokenize your actual traffic with each model’s own tokenizer. Second, anything that inflates token counts — verbose formats, redundant whitespace, a language the tokenizer handles poorly — silently inflates your bill and shrinks your effective context.

Why characters lose: the quadratic bill

The deeper reason character-level models struggle is arithmetic. Self-Attention compares every position with every other position, so its cost grows with the square of sequence length n — the O(n²) term you will meet properly in the attention module. If switching from subwords to characters makes the same text several times longer in positions, the attention cost multiplies by the square of that factor, and generation needs several times more sequential steps for the same output.

On the other side of the trade, every vocabulary entry is a row in the Embedding matrix and a column in the Unembedding projection. A word-level vocabulary of hundreds of thousands of entries spends an enormous share of the parameters on rows that rarely fire, and its final Softmax must score every one of them at every step. Subword vocabularies sit in the valley between these two costs: sequences short enough for attention, vocabularies small enough for the embedding and output layers. Character- and byte-level architectures remain an active research direction precisely because they attack the sequence-length side of this trade rather than accepting it.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.