The Backup That Wasn't
We had backups. Daily snapshots to S3. Perfectly configured. Never tested.
When we needed to restore after a data corruption incident, we discovered the backups had been silently failing for 3 weeks. The S3 bucket policy had changed, and nobody noticed.
Never again.
Rule 1: Test Your Restores







