Running large language model (LLM) inference at scale typically forces a KV cache trade-off: you either pay for oversized GPU instances to accommodate a growing KV cache, or you accept slow time-to-first-token (TTFT) as identical prompts get recomputed on every request. For teams deploying a broad catalog of publicly available foundation models (FMs), such as Qwen, Llama, DeepSeek, and others, across per-business-line endpoints, Retrieval Augmented Generation (RAG) pipelines, or multi-turn dialogue applications, this trade-off translates directly into higher infrastructure cost and degraded user experience.

The root cause is straightforward. During generation, vLLM stores the attention keys and values for every token it has already processed in a KV cache, so it doesn’t recompute them on each step. Prefix caching extends this by reusing that cache across requests that share the same leading tokens (like a common system prompt). On cost-efficient instances like ml.g6e.4xlarge (48 GB per GPU), once model weights and runtime allocations are accounted for, the memory left for prefix caching is limited, and it tightens further with larger models or higher concurrency. Cache hit rates drop on long prompts, identical system prompts get re-prefilled on every request, and horizontally scaled vLLM replicas each maintain isolated caches. Routing to a different replica is functionally a cold start.