TL;DR— Inference cost conversations obsess over FLOPs and token prices, but the real constraint on LLM serving is memory bandwidth— specifically the cost of moving the KV cache in and out of GPU memory on every decode step. Teams that optimize for compute utilization instead of memory traffic end up overpaying for capacity they never use. The fix is architectural: disaggregating prefill from decode, right-sizing batch and context, and treating bandwidth as the scarce resource it actually is.
Every infrastructure conversation about running large models starts with FLOPs. GPU spec sheets lead with them. Cost calculators multiply them against price-per-hour. Vendors brag about them. And almost none of it matters for the workload most teams actually run: autoregressive decoding at scale.
The thesis here is simple and under-discussed: for the majority of production LLM inference, the bottleneck isn't compute. It's memory bandwidth. Specifically, it's the cost of shuttling the key-value cache in and out of high-bandwidth memory on every single decode step. If you're sizing infrastructure, negotiating vendor contracts, or designing a serving stack around FLOPs, you're optimizing the wrong variable, and you're probably overpaying.
