It Started With a Disk Usage Alert
While going through our monitoring dashboards, I noticed that one of our backup object storage servers had been experiencing high disk usage for quite some time. Since the issue had remained unresolved, I decided to take ownership of the investigation.
Our existing disaster recovery setup consisted of two independent single-node OpenStack Swift deployments. Instead of native replication, data was periodically copied from the primary server to the backup server using a custom synchronization process, where objects were downloaded, staged on the backup server, and then uploaded back into its own Swift instance.
After reviewing the existing architecture and previous discussions, I found that several approaches had already been proposed. These included using rsync or rclone for more efficient synchronization, inotifywait for near real-time replication, and even introducing a Virtual IP (VIP) with Keepalived to improve failover. Each proposal addressed a specific operational challenge and aimed to improve disaster recovery.
The more I read, however, the more I realized that we were continuously adding new components and operational logic around keeping two completely independent object storage systems in sync.
