We run an AI companion bot. Every chat turn, the model sees the same ~5K-token prefix — character persona, content-tier rules, formatting guardrails, a memory blob — plus one new user line. Without caching, we pay for those 5K input tokens every single turn. So we turned on prompt caching across the providers we route through, measured it, and the spread was bigger than any of the marketing pages prepared us for.
Here's the table that survived four weeks in production, plus the one gotcha that ate two weeks before we figured it out.
The hit-rate table
Provider / model
Hit rate
