How an NLB Health Check, a CloudWatch Blind Spot, and an Outdated Requirement Caused a 14-Hour Production Incident.
AWS infrastructure migration failed silently: NLB health checks couldn't send custom Host headers required by ALB security rules, causing nodes to fail checks and drop traffic. CloudWatch 1-minute averages masked rapid oscillations—a stark lesson that infrastructure monitoring must watch data plane reality, not smoothed control plane metrics.
Originally published on my Hashnode blog. This version has been adapted for the DEV community.
Confidentiality Notice: To protect intellectual property, certain architectural values, network configurations, and specific company identifiers have been generalized or obfuscated.
ALB Targets: Healthy.
NLB Targets: Healthy.
ECS Tasks: Running Normally.
It was a Tuesday afternoon. Nothing felt different about the deploy. Same process as always. Green...
A survival guide for when everything goes wrong in production. The pipeline is green. Every stage...
My AI told me the job was done. My container said it was running. My firewall said it was up. All...
A real-world lesson from a production-like AWS lab challenge The Scenario That Should Scare...
Load balancers are the silent infrastructure. You don't think about them until they start dropping...
Every dashboard was green and zero blog posts went live. Exit codes tell you the job ran, not that the outcome happened. Here is…
