Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup.
Whether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my LLM benchmarking guide. Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle.
What prefix caching reuses
During prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache.
This is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests.
