An expired cert took down a subdomain on a Saturday with zero alerts. Here is the bash sweep that catches the failures hiding in plain sight.
A TLS certificate on a random internal subdomain expired on a Saturday. We had great uptime checks on the main app, but nobody thought to monitor the auth subdomain.
No logs screamed, no exceptions fired, no deploys failed. The certificate just quietly hit its expiration date and stopped being valid. The first alert we got was a frustrated Slack message with a screenshot of Chrome's full page security warning.
It was a completely stupid outage, which is exactly why it's worth talking about. As an industry we obsess over things that fail loudly. We set up complex alerting for 500s, OOM kills, and pod restarts. Then we completely ignore the infrastructure that fails by simply hitting a deadline.
The silent killers
