A team that produces regulatory documents kept getting the same kind of question from other teams: does the current rule allow X? Answering meant someone reading through memos, manuals and regulations to find where the rule was stated, then writing a summary with the source cited. The work was real, and the answer was almost always already written down somewhere.
Retrieval fits this exactly. The user asks, the system summarises, and it cites which document the answer came from. The citation is the whole point — a summary without a source is useless when the question is about a rule.
It also fits the constraint from Part 1: the documents can't leave the perimeter, there's no GPU, and each generated answer costs tens of seconds on CPU.
Which is why a semantic cache looked like the most obvious optimisation in the system. Several teams ask about the same regulation. If two questions mean the same thing, serve the stored answer and skip the model entirely. I implemented it with cosine similarity between question embeddings and a threshold of 0.92, and it went out in the reference implementation this repository publishes — always on, with no flag to turn it off.
Then Giulio D'Erme asked me to prove that the threshold was safe. His example was clinical rather than administrative — pacientes con fiebre against pacientes sin fiebre, one token apart, opposite correct answers — but it transfers to every corpus I care about.
