A regression came in for our German enterprise users on the support agent. Quality had dropped for that one cohort, and I opened the trace store expecting to find the problem in a couple of minutes. We had tracing. I had set it up myself.
What I got was a flat list of 28 spans. None of them carried the prompt version. The model-call spans were named three different things across the same service, because different libraries named them differently. The retrieval spans had the raw user query sitting in them as plain text. And one span had a four-kilobyte blob holding the entire prompt body.
Forty-five minutes in, I still had not found the regression. The app was traced. It was not traced in any way that helped.
That incident is why I rewrote how we trace. Here is what actually makes a trace useful when you are the one staring at it at 2 AM, no code, just the shape of it.
What "a good trace" actually means
