Originally published on hexisteme notes.
Prompt caching is the largest single token lever for any long-lived LLM session — a cached token is served at roughly a tenth of its normal price. It is also one of the easiest things to break by accident, because when it breaks there is no error. One timestamp in the system prompt, one unsorted JSON key, and your cache-read rate quietly goes to zero while the bill goes up tenfold. This is how the failure works, and how to make it loud.
A prompt cache is a prefix match: the cache key is the exact bytes of the rendered prompt up to a breakpoint, and a single byte change anywhere in the prefix invalidates everything after it. So anything volatile near the front of the prompt — a datetime.now() in the system header, a per-request UUID, a dict serialized without sorted keys, a tool list that reorders — silently defeats caching for the entire request. The cost is real (a miss is ~10× a hit on the cached span) and invisible (no exception, no warning). The only signal is one field in the usage object, and the only durable fix is to assert on it.
How prefix caching actually works
The mental model that prevents every bug below is one sentence: the cache key is the bytes of the rendered prompt, matched as a prefix. The provider renders your request in a fixed order — tools, then the system prompt, then the messages — hashes the bytes up to each cache breakpoint, and looks for a prior entry with the same prefix. If it finds one, the shared span is a cache read; the rest is processed fresh. If the very first byte differs, nothing matches and the whole thing is processed at full price.
