The short version: an LLM server that loads fine and answers your test prompt will still crash under real traffic, because GPU memory for inference is dominated by the KV cache — per-request memory that grows with context length and concurrency, not by the model weights. The fix that keeps a deployment alive isn't a bigger card; it's admission control: deciding, before you accept a request, whether there is enough KV-cache budget to finish it. This post is how to compute that budget and where to enforce it.
I'm writing this as a follow-up after a commenter on an earlier post about GPU memory made a sharp point: the first deploy should be budgeted around admission control, not just whether the weights load, and a single long-context request is often a better stress test than a pile of short chats. That's exactly right, and it deserves its own walkthrough.
Why does a server that "fits" still crash under load?
Loading the weights proves the model fits. It says nothing about whether the server serves. During inference, the memory that actually decides survival is the KV cache: the cached key/value tensors for every token already in a request's context, so the model doesn't recompute them each decode step. Its size per request is roughly:
