Most inference optimization content is written for people running fleets of H100s. If you're serving models out of a single GPU box, a modest EC2/ECS setup, or even CPU inference for smaller models, a lot of that advice doesn't transfer — the constraints are different, and the "just add more GPUs" answer isn't available to you.
This post covers the three levers that actually move the needle at small-to-mid scale: KV cache management, quantization, and the latency tradeoffs that come with both. No cluster-scale assumptions.
Why this matters before it's a crisis
Inference cost and latency don't show up as a line item until they do — usually right around the point where a side project or an MVP gets real traffic. At that point you have three options: spend more on hardware, spend more on hosted API calls, or actually understand what's eating your latency and memory budget. The third option is the only one that scales your understanding along with your infrastructure.
Lever 1: KV cache — what it is and why it dominates memory
