Speculative decoding: when and why it actually speeds up inference
Your chat endpoint serves 200 requests per second. The model is a 70B Llama 3 fine-tune. The GPU is sitting at 78% utilization, but the user-facing latency is still bad — 380 ms to first token on the median request, 1.1 s P99. The naive read is "we need a bigger box." The actual read is that the GPU is memory-bound, not compute-bound: most of the time is spent shipping weights and KV-cache state from HBM into the SMs, one token at a time, waiting for the next one. Speculative decoding is the technique that turns that one-token-at-a-time pipeline into a several-tokens-at-a-time pipeline without changing what the model actually samples. In our case it dropped p50 TTFT from 380 ms to 140 ms with the same hardware and the same 70B weights.
Here's what it is, what the variants are, and when it stops being a free lunch.
Why this matters in practice
The throughput ceiling for an autoregressive LLM on a single GPU is set by the cost of moving one token's worth of logits and the next token's worth of attention state, not by FLOPs. Doubling the model's parameters roughly doubles the time-per-token on a memory-bound workload, but it does not double the FLOPs the SMs can do — the SMs are sitting idle. Speculative decoding addresses this by doing the heavy forward pass over the target model only every K tokens, and filling the gaps with a much smaller draft model that proposes K tokens in the time the target would have done one.
