Speculative decoding: draft cheaply, verify in parallel
Generating N tokens sequentially requires N passes through a large target model. Speculative decoding uses a fast, small draft model to guess K candidate tokens cheaply, then runs the large target model once in parallel to verify all K tokens simultaneously. Accepted tokens are emitted immediately; rejections cost only one target pass.
Batched Verification
The large model evaluates K draft tokens in a single forward pass (which takes nearly the same time as generating 1 token due to bandwidth bottlenecks).
Lossless Generation
A modified rejection sampling algorithm ensures the output probability distribution matches the target model exactly — zero quality loss.
Acceptance Rate Dependence
Speedup is directly driven by how often the target accepts draft tokens (α). High α yields 2-3x latency reduction.
If a draft model guesses 4 tokens with a 75% acceptance rate (α=0.75), the target model emits an average of 2.73 tokens per pass, achieving ~2x wall-clock speedup.
- Medusa and Eagle use specialized multi-head draft heads on top of the base model rather than separate small models, avoiding extra VRAM overhead.
- Speculative decoding is deployed in high-throughput coding assistants and chat endpoints where low latency is critical.