TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built a dynamic few-shot selector that retrieved the eight nearest labeled examples for each input, and we built its index out of the same labeled_tickets.jsonl the eval set was sampled from. So for every eval case, the nearest neighbor in that index was the eval case itself, gold label attached, pasted into the prompt directly above the question we were about to ask. The model was not answering. It was copying. Measured against tickets the index had never seen, real accuracy was about 0.79. The reframing that stuck for me: contamination is not just a training-time problem you inherit from a model vendor. Any pool you draw prompt content from is part of your eval's input. If your eval set and your few-shot pool share a parent file, the leak is yours, in your repo, shipped by someone on your team last Tuesday. Split the pools by content hash at the source, then check the prompt and not only the training data.
The trace that ruined a good number
I was not looking for this. I was chasing p99 latency on the routing endpoint, which had crept past two seconds and was making the queue back up. So I pulled a slow trace and started reading the prompt we actually send, top to bottom, the way you do when you suspect somebody stuffed too much into the context window.
