Decoding and Sampling

Greedy, beam, temperature, top-k, top-p: how one token actually gets chosen, and how to pick settings you can defend.

Content last verified 2026-09.

Lessons

1. Greedy Decoding and the Beam That Faded
2. The Sampling Knobs
3. Penalties, Stops, and Constraints
4. Choosing Settings You Can Defend

Sources

• Holtzman et al. (2020) — The Curious Case of Neural Text Degeneration (nucleus sampling)
• Fan, Lewis & Dauphin (2018) — Hierarchical Neural Story Generation (top-k sampling)