The page came in at 2:14 a.m., which is the only time pages ever come in. I have a theory that PagerDuty holds them in a little queue until it's certain you're in the deepest part of your sleep cycle, and then it releases them all at once, like a cat knocking a glass off a table while making eye contact. I've been doing this work for twenty-five years — long enough to have carried a literal pager at Yahoo, long enough to have learned resilience the hard way at Netflix, long enough that a 2 a.m. alert no longer produces adrenaline so much as a kind of tired affection. Here we go again. Let's see what you've got.
What it had was a payments-adjacent service that had stopped serving. Not slow. Not flapping. Stopped. The kind of outage where the graph doesn't decline, it just falls off the edge of the world.
This is the first in a series I'm calling Troubleshooting Kubernetes — war stories from the cluster, with the actual fixes, told by someone who genuinely loves this work and is also deeply, professionally tired. Those two things aren't in tension. They never have been.
The symptom, and the small lie it told
First move, always the cheapest one: look at the pods.
