Years ago at RightScale I learned more about distributed systems from broken log files than from any design doc. A request came in the front door, fanned out through a workflow service, hit a plugin, and the plugin called some cloud API. When it failed, the only way to find where was to line up the logs of every service it passed through. So we threaded a trace ID from the frontend all the way to the cloud call and back. With it, a failure was a search query. Without it, you were guessing.
the lesson came back
It stuck. I just had to learn it twice.
Building finfocus, a FinOps CLI that talks to cloud providers through gRPC plugins, I hit the same wall. Except this time there was a second reader who couldn't follow the logs: the agent. I'd ask Claude to find why a plugin call failed, and it would dig through the finfocus logs, reach the gRPC boundary, and find nothing on the other side. Unconnected traces, or no traces at all. I'd reached for zerolog early, but I'd wired it up wrong. The CLI logged. The plugin logged. Nothing tied the two together, so neither of us could see across the boundary.
Once the trace ID actually crossed that boundary, troubleshooting got fast. The agent could make a call, follow it from the finfocus CLI through the gRPC plugin and back, and tell me which side broke. Bug hunting went from a séance to a grep.
