Last time I told you to bring coffee. I did. This one didn't start at 2 a.m. — it started at 10:40 on a perfectly ordinary Tuesday, which, if anything, is worse. There's a particular kind of dread that only arrives in full daylight, when you're well-rested and caffeinated and therefore have no excuse for not understanding what's happening. And I did not understand what was happening.
Welcome back to Troubleshooting Kubernetes. Today's episode is about the most unsettling category of outage there is: the one where nothing is broken. Every light is green. Every pod is Running. Every dashboard is calm. And your users are getting 500s.
I've been doing this long enough — a couple of decades and change, across a few places I've been lucky to work — to have developed a healthy distrust of green. Green is not a fact. Green is a claim. And on that Tuesday, something was lying to me with a very straight face.
The symptom that wasn't there
The error rate on the checkout API had crept from basically-zero to about 18% over twenty minutes. Not a spike. A creep — the kind that makes you wonder if it's real or if the graph is just having feelings. So, first move, always: is this real, or is the monitoring lying?
