You built the system the way everyone tells you to.
Three API servers instead of one, so a crash doesn't take you down. A load balancer in front of them spreading traffic around. Your domain points at the load balancer, the load balancer health-checks the backends, and when one server dies the other two pick up the slack. Textbook.
Then at 2 AM the load balancer itself dies.
Not a backend. The balancer. Maybe the box lost power, maybe the kernel panicked, maybe HAProxy hit an out-of-memory kill. It doesn't matter which. Your three API servers are sitting there completely healthy, idle, ready to serve, and every single request is timing out. Because nothing can reach them anymore.
You removed the single point of failure from your application tier and quietly moved it to the thing in front.
