Tokenization: Text Becomes Tokens

Why models read subword tokens instead of words, how byte-pair encoding builds a vocabulary, and the artifacts every engineer eventually hits.

Content last verified 2026-09.

Lessons

1. Why Models Read Tokens, Not Words
2. Byte-Pair Encoding, Step by Step
3. Tokenizer Families and Trade-offs
4. Tokenizer Artifacts and Gotchas

Sources

• Sennrich, Haddow & Birch (2016) — Neural Machine Translation of Rare Words with Subword Units (BPE)
• Kudo (2018) — Subword Regularization (unigram language model tokenization)
• Kudo & Richardson (2018) — SentencePiece: A simple and language independent subword tokenizer
• Radford et al. (2019) — Language Models are Unsupervised Multitask Learners (GPT-2, byte-level BPE)