A 70B model generates one token per forward pass, and each pass reloads weights from VRAM, computes attention across the context, and synchronizes memory. Between tokens, the GPU sits idle while it waits for sequential dependencies to resolve.
On an H100, a 70B model produces one token every 30-50ms. The GPU has enough compute capacity to process multiple tokens in parallel, but the sequential dependency prevents it — each token depends on the previous one, and the pipeline stalls.
Speculative decoding breaks that bottleneck by letting you generate multiple tokens in the time it normally takes to generate one, without changing the output distribution. The tokens you get are statistically identical to what you'd get from standard autoregressive decoding; the only difference is how fast you get them.
This guide covers the mechanics, the variants available in 2026, acceptance rate tradeoffs, and practical setup across llama.cpp, vLLM, SGLang, and TensorRT-LLM.
How Autoregressive Decoding Works (and Why It's Slow)