Tokenizer Families and Trade-offs
Lesson 3 of 4 in Tokenization: Text Becomes Tokens.
Every mainstream tokenizer chases the same goal — cover any text with a fixed vocabulary of useful pieces — but four families get there differently, and the differences show up in the tokens you see.
BPE you already know: greedy frequency merges, bottom-up. WordPiece — the tokenizer behind BERT and Google’s earlier translation systems — looks nearly identical but picks each merge by a likelihood criterion (roughly: prefer the pair that most improves a language-model score of the corpus, not just the raw count), and marks word-continuation pieces with a ## prefix. Unigram tokenization (Kudo, 2018) runs in the opposite direction: start from a huge candidate vocabulary, fit a probabilistic model treating each word as generated from independent pieces, and repeatedly prune the pieces whose removal hurts corpus likelihood least. Because it is probabilistic, one string can have several plausible segmentations — useful in training, where sampling alternate splits acts as regularization. Byte-level BPE (popularized by GPT-2) is ordinary BPE with the 256 byte values as the base alphabet, so no character in any script — or emoji, or binary junk — is ever out of vocabulary.
Two names you will meet are libraries, not algorithms: SentencePiece (Kudo & Richardson, 2018) implements BPE and unigram while treating input as a raw character stream — whitespace becomes an ordinary symbol (rendered ▁), so the tokenizer needs no language-specific pre-splitting and works for scripts that do not use spaces. tiktoken and the Hugging Face tokenizers library are fast implementations of byte-level BPE and friends.
| Family | How the vocabulary is built | Unknown text? | Signature trait | Seen in |
|---|---|---|---|---|
BPE | Bottom-up: greedily merge the most frequent adjacent pair | Impossible only if base alphabet covers the text; character-based variants can still hit unknowns | Deterministic ordered merge list; splits mirror corpus frequency | The original subword NMT models; ancestor of most modern LLM tokenizers |
Byte-level BPE | Same merge loop, but the base alphabet is the 256 bytes | Never — any string is bytes, so coverage is total by construction | No unknown token needed; unfamiliar scripts fragment into many byte pieces | GPT-2 and many successors |
WordPiece | Bottom-up like BPE, but merges chosen by likelihood gain rather than raw frequency | Falls back to an unknown token for uncovered characters |
| BERT and its family |
Unigram (SentencePiece) | Top-down: start huge, prune pieces that matter least to corpus likelihood | Byte or character fallback, per configuration | Probabilistic — multiple segmentations exist; sampling them regularizes training | T5, many multilingual and translation models |
Whatever the family, one dial dominates the design: vocabulary size. Turn it up and average tokens get longer, so the same text needs fewer of them — sequences shrink, more meaning fits per Context window slot, and per-token costs stretch further. But every added entry is a new row in the Embedding matrix and a new logit the output layer must score at every step, and the rarest entries are seen so seldom in training that their vectors stay poorly learned. Turn the dial down and the parameters concentrate on well-trained pieces — while every sequence gets longer and attention pays its quadratic tax on the difference.
There is no universally right setting — only a fit between vocabulary, training corpus, and the languages and domains the model must serve. Multilingual models in particular tend to want larger vocabularies, because their token budget has to stretch across many scripts at once.
Key terms: Tokenizer, Subword, Byte-pair encoding (BPE), WordPiece, Unigram tokenization, Byte-level BPE
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.