Variants, and When It Helps
Lesson 3 of 3 in Speculative Decoding.
The classic two-model setup is only one way to get drafts, and running a second model has real costs: extra GPU memory for its weights and KV cache, a deployment to operate, and the vocabulary-compatibility constraint from lesson one. A family of variants attacks exactly that overhead — the names below are worth recognizing, described qualitatively.
Self-speculation uses the target model itself in a cheaper configuration as its own draft — for example, running only a subset of its layers (early exit) or an otherwise reduced path — so draft and target share weights and no second model is deployed. Multi-head drafting, in the spirit of the widely used Medusa approach, bolts small extra prediction heads onto the target so a single forward pass proposes several candidate continuations, often organized as a tree of alternatives verified together. Lookup-based drafting needs no neural draft at all: when output copies from the context — extraction, editing, quoting retrieved documents — candidate continuations can be pulled by matching n-grams in the prompt. Each variant changes where drafts come from; verification, and the exactness argument behind it, stays the same.
Serving engines treat all of this as configuration, not architecture: widely used open projects such as vLLM, TensorRT-LLM, SGLang, and llama.cpp ship speculative decoding as a feature you enable and tune — pick a draft source, set the draft length k, measure. Which brings us to the real question: measure what, on which workloads?
| Workload | Expected benefit | Why |
|---|---|---|
Code completion, boilerplate generation | High | Syntax and idiom make the next token highly predictable, and these workloads run greedy or near-greedy — acceptance stays high, spans run long. |
Structured output (JSON, forms, templates) | High | Much of the output is dictated by the format: braces, keys, delimiters. Even a weak draft agrees with the target on forced tokens. |
Extraction, editing, RAG answers quoting sources | High | Output copies spans from the context — the easiest text to draft. This is where lookup-based drafting is at its best. |
Low-temperature factual chat | Moderate | Concentrated distributions keep acceptance decent, but free-form prose has more genuine forks than code or templates. |
High-temperature creative writing | Low | Hot sampling flattens both distributions; the draft rarely proposes the token the verifier favors. Acceptance falls, and draft overhead can exceed the win. |
Throughput-saturated batch serving | Low, possibly negative | At large batch sizes decode stops being memory-bound — the GPU’s spare compute is already spent on other requests, so verification FLOPs now compete with batch throughput. |
The two “low” rows share one root cause: speculative decoding spends the GPU’s idle arithmetic — the compute left stranded while decode waits on memory. Anything that has already claimed that headroom shrinks the free lunch. High Temperature attacks the other input: it lowers acceptance, so fewer drafted tokens survive each verify pass.
The batching interaction deserves special respect, because Continuous batching is the default posture of every modern Serving engine (Continuous Batching and Serving Engines covers it). Batching many requests into each decode step is also a way to convert idle compute into useful work — and at high load it is usually the better one, because it needs no acceptance luck. That makes speculative decoding primarily a latency tool for interactive, low-batch traffic where TPOT is what users feel, and a poor throughput tool for saturated offline serving where tokens per second per GPU is the meter. Engines let both features coexist; the point is to know which meter you are optimizing before flipping flags.
So the operator’s checklist is short: enable it on a copy of real traffic, read the acceptance rate the engine reports, compare TTFT, TPOT, and Throughput against the baseline, and keep it only where the numbers earn it. A low acceptance rate is not a failure of the technique — it is the technique telling you this workload is not predictable enough to draft.
In production
Acceleration features move your latency and cost meters — they must never move the output contract. But “the distribution is unchanged” is a theorem about the algorithm, not a warranty for every implementation between you and the GPU. Wherever a platform or engine claims output-identical acceleration, verify it against your own traffic: fix a set of prompts, run greedy decoding with the feature on and off, and diff the tokens.
AWS
On AWS the flag lives at two altitudes. Self-hosting on EC2 or EKS with an open engine (vLLM, TensorRT-LLM, SGLang), speculative decoding is your configuration to enable, tune, and measure — draft choice and draft length k included. On managed platforms like Amazon Bedrock or SageMaker endpoints, acceleration options may apply optimizations of this family under the hood; treat any latency-optimized mode as a meter change to validate, and re-run your output-equivalence and quality checks whenever you toggle one.
Azure
The same split applies on Azure: managed model deployments in Azure AI Foundry may bundle serving-side accelerations you do not directly control, while self-hosted engines on AKS expose speculative decoding explicitly. If you plan capacity in provisioned-throughput terms, remember the speedup is workload-dependent — a benefit measured on code completion does not transfer to a creative-writing endpoint, so benchmark each traffic profile separately before committing capacity numbers.
Google Cloud
On Google Cloud, Vertex AI managed serving and self-managed engines on GKE follow the same pattern: the closer you are to the engine, the more explicitly you choose and tune the draft. Whichever altitude you operate at, make acceptance rate a first-class dashboard metric next to TTFT and TPOT — it explains speedup changes that would otherwise look like infrastructure noise, such as a prompt-template change quietly making your traffic less predictable.
Key terms: Speculative decoding, Draft model, Decode, Greedy decoding, Temperature, Continuous batching
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.