The Backup That Wasn't

We had backups. Daily snapshots to S3. Perfectly configured. Never tested.

When we needed to restore after a data corruption incident, we discovered the backups had been silently failing for 3 weeks. The S3 bucket policy had changed, and nobody noticed.

Never again.

Rule 1: Test Your Restores