The Trick: Draft, Then Verify

Lesson 1 of 3 in Speculative Decoding.

Decode is a queue of one. Each generation step runs a full forward pass — streaming essentially all the model’s Weights through the GPU — and all that traffic buys you a single Token. The arithmetic units spend most of the step waiting on memory, which is why time-per-output-token (TPOT) is dominated by bandwidth, not math (Prefill and Decode builds this picture; Pope et al. 2022 is the standard reference for the memory-bound regime).

Hiding inside that bottleneck is a strange asymmetry: checking text is much cheaper than writing it. Give a transformer five tokens and it can score all five positions in parallel in one pass — that is exactly what Prefill does. The weights make the same trip through memory whether the pass scores one position or five, so in the bandwidth-bound regime the extra positions ride along almost free. Generation is expensive only because each new token needs its own serial trip.

So the question becomes: what if something cheap wrote a plausible next few tokens, and the big model only had to check them?

That is Speculative decoding, proposed independently in two papers — Leviathan, Kalman and Matias (2022, arXiv:2211.17192) at Google and Chen et al. (2023, arXiv:2302.01318, who call it speculative sampling) at DeepMind. The loop has three moves:

  1. Draft. A small, fast Draft model — typically a much smaller model from the same family, sharing the target’s Tokenizer and Vocabulary — runs k cheap serial steps and proposes the next k tokens.
  2. Verify. The target model runs one forward pass over all k proposals at once, producing its own probability distribution at every drafted position — parallel, like a tiny prefill.
  3. Accept or reject. Compare positions left to right. The accepted prefix is committed. At the first rejection, everything after it is discarded and that position is resampled from a corrected distribution (next lesson). If all k survive, the verify pass has one more gift: the target’s distribution for position k+1 is already computed, so you sample a bonus token for free.

Net effect: every expensive target-model pass now commits between 1 and k+1 tokens instead of exactly one. The floor is 1 because even a first-position rejection is resampled into a committed token — a cycle never comes back empty-handed.

One speculation cycle: draft, verify, accept or reject

  1. Context so far

    The prompt plus every token committed by earlier cycles. Both models keep their own KV cache over it.

  2. Draft model proposes k tokens

    k fast serial decode steps on the small model — cheap because every step is small, not because it is parallel.

  3. Target model: one pass over all k

    A single forward pass scores every drafted position in parallel, like a tiny prefill, yielding the target’s distribution at each one — plus the distribution for the position after the last draft.

  4. Any position rejected?

    Walk left to right, applying the acceptance rule at each drafted token. The first rejection ends the walk.

  5. Commit all k + 1 bonus token

    Every draft survived, and the verify pass already computed the target’s distribution for the next position — sample it at no extra cost.

  6. Commit accepted prefix + corrected resample

    Tokens after the rejection are discarded. The rejected position is resampled from an adjusted target distribution, so even this path commits at least one token.

  7. Stop token or length limit?

    Committed tokens stream out exactly as in ordinary decoding — the client never sees the speculation.

  8. Response complete

    Same tokens the target model would have produced alone — just fewer serial passes to get there.

Where does the win come from, on the meter? Every accepted token is a serial target-model pass you never ran. The verify pass is not literally free — its FLOPs grow with k — but while decode is memory-bandwidth-bound and k is small, its wall-clock cost stays close to a single decode step, because the dominant cost (weight traffic) is paid once either way. The draft is not free either: k small-model steps per cycle, plus the draft’s own KV cache sitting in GPU memory.

Whether the trade nets out positive depends on how often the target agrees with the draft — the acceptance rate — and on how cheap the draft is relative to the target. In their own experiments, Leviathan et al. (2022) report roughly 2–3× wall-clock speedups on translation and summarization models, and Chen et al. (2023) report roughly 2–2.5× on a 70B-parameter model in a distributed setup; treat both as what they are, results for specific model pairs on specific workloads, not a promise for yours.

And the headline property, the one the next lesson proves: the committed tokens are not an approximation. They are distributed exactly as if the target model had generated every one of them itself, one expensive step at a time. This is why serving stacks can ship speculative decoding as an invisible acceleration flag rather than a quality trade-off.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.