After all the hype around tokenmaxxing, we have finally realised something that was hiding in plain sight: every LLM request comes at a cost. This becomes even more of a challenge when enterprises start taking their AI PoCs to production and first encounter system design’s most fundamental problem: scale. Repeated questions can quickly turn into a massive cloud bill.
But in my learning journey, I have noticed this: Customers rarely ask the same question with the exact same wording. For example, in an e-commerce support chatbot, the most common questions would be: "Where is my order?", "Can you track my package?", "Has my shipment been dispatched yet?", "When will my order arrive?” - All seemingly different prompts, but all seeking the same information.
Great, so why not just cache the most requested information? Well, a traditional cache only works if the request is identical. Change a few words, and it's a cache miss; resulting in another expensive LLM call.
One solution I could think of and ended up building was a Go library for semantic LLM caching that combines deterministic lookups with vector similarity search.
Link to my repo: https://github.com/Suraj370/semantic-cache-for-llm
