An LLM request is two workloads in a trench coat — a heavy, bursty prefill and a stream of tiny latency-sensitive decodes. Running them on the same engines lets one big prompt stall everyone. Splitting them fixes it.
TL;DR: Every LLM request is two very different jobs. Prefill reads the whole prompt — heavy, bursty, and slow for long contexts. Decode then emits tokens one at a time — tiny, but latency-sensitive. Run them on the same engines and a big prefill jumps ahead of everyone's decodes: head-of-line blocking, and the token stream stutters. Prefill/decode disaggregation puts prefill and decode on separate pools so decodes never queue behind a prefill. In a runnable Go simulation, splitting the pools cut p99 inter-token latency by 66% (88ms → 30ms) — trading a little time-to-first-token for a far smoother stream. This is now standard practice in frontier serving stacks (DistServe, Splitwise, vLLM × Mooncake).
Mental model: a coffee shop with one worker who both grinds beans and pours espresso. A customer orders a giant batch grind and everyone waiting for a simple pour is stuck behind it. Split the shop into a grinder station and a pour station and the pours keep flowing no matter how big the grind.
