You ship an LLM service. p50 latency looks great. Then a user pastes a 40-page contract into the chat, and for the next 400 milliseconds every other user's tokens stop arriving. Their streams freeze, then catch up in a burst. Your dashboards show inter-token latency spikes with no obvious cause. Nothing crashed. Nothing is rate-limited. One long prompt did it.
This is prefill-decode interference, and the fix — chunked prefill — is one of the highest-leverage knobs in an LLM serving stack that almost nobody tunes deliberately. Here is the mechanism and the config.
TL;DR
Prefill is compute-bound and runs in one giant forward pass; decode is memory-bound and runs one token at a time. A naive scheduler runs a long prefill as a single batch step, and every in-flight decode request stalls until it finishes.
Chunked prefill splits the prompt into fixed-size chunks and interleaves them with decode tokens in the same forward step, bounding step time so decode latency stays smooth.
