From N-grams to Transformers

Seventy years of language modeling in one arc: counting words, the neural turn, attention, and the scale era — with the dates and papers that mattered.

Content last verified 2026-09.

Lessons

1. N-grams: Language by Counting
2. RNNs and the Neural Turn
3. Attention Arrives
4. Scale and the LLM Era

Sources

• Shannon (1948) — A Mathematical Theory of Communication
• Bengio, Ducharme, Vincent & Jauvin (2003) — A Neural Probabilistic Language Model
• Sutskever, Vinyals & Le (2014) — Sequence to Sequence Learning with Neural Networks
• Bahdanau, Cho & Bengio (2015) — Neural Machine Translation by Jointly Learning to Align and Translate
• Vaswani et al. (2017) — Attention Is All You Need