Speculative Decoding

Draft cheap, verify exact: how a small model makes a big one faster without changing a single output token.

Content last verified 2026-09.

Lessons

1. The Trick: Draft, Then Verify
2. Why the Output Stays Exact
3. Variants, and When It Helps

Sources

• Leviathan, Kalman & Matias (2022) — Fast Inference from Transformers via Speculative Decoding
• Chen et al. (2023) — Accelerating Large Language Model Decoding with Speculative Sampling
• Pope et al. (2022) — Efficiently Scaling Transformer Inference