Nobody touched the model. Nobody touched the traffic. Nobody touched the prompts, as far as anyone could tell from the diff. And yet the API bill for our agent tripled between one deploy and the next, and average response latency to first token nearly doubled with it.
The model was fine. The prompt cache was dead — and it had been killed by a single line we added to be "helpful."
How prompt caching actually works
Most people treat prompt caching as a magic discount switch: turn it on, get cheaper calls. What it actually does is match a byte-for-byte identical prefix of your request against a cache from a previous call, up to a breakpoint you (or the SDK) declare. If the prefix matches exactly, the provider skips re-processing those tokens through the model's attention layers and charges you a fraction of the price for them — often around 10% of the input cost, with a large cut in time-to-first-token too, since the KV cache for that prefix is already computed and sitting in memory.
The keyword is exact. Not "semantically similar." Not "mostly the same." One different token anywhere before the breakpoint, and the entire prefix after that point stops matching — even if 99% of it is identical to the last call.
