Three years ago we had our first real outage. Six hours of downtime. Thousands of angry users. Multiple executives on the call. Here's what we did right, what we did wrong, and what we'd do differently.
What we did right
1. Communicated immediately. The moment we knew we had a problem, we updated the status page and emailed our biggest customers personally. Not when we had answers. When we had a question.
2. Had a single incident commander. One person making calls. Not a committee. When the CEO tried to direct technical work, the IC politely rerouted and told her where her help was actually needed (talking to customers).
3. Took care of our people. During hour 4, I ordered food. During hour 5, I forced the primary engineer off the call for 20 minutes to walk outside. Long incidents destroy people. You have to feed them and force them to rest.
