The Problem We Were Actually Solving
We needed a staging environment that could tolerate the stupidity of humans. Not a toy cluster that looked like production but couldnt survive a mis-typed curl flag. The real requirement was: if an engineer turns staging into a dumpster fire at 3 am, nothing outside staging should notice. We also needed to ship a new subscription checkout flow for creators in countries where PayPal blocks transactions. The flow had to store settlement schedules, retry failed charges, and emit events to Kafka so analytics could bill creators in USD without touching the blocked jurisdiction. The first cut used DynamoDB with on-demand capacity and TTLs, but the finance team vetoed it because the eventual-consistency model could under-charge a creator in Kazakhstan by 0.03 USD and we wouldnt know for 12 hours.
What We Tried First (And Why It Failed)
We started with a Terraform module for RDS Postgres 14, parameter group set to db.t3.medium, publicly accessible = true, and storage_encrypted = false. It passed the linting stage because the linter only checked for AWS tags. We deployed staging with terraform apply -auto-approve -var environment=staging. Two weeks later an intern ran a chaos experiment that killed the master node; Prometheus screamed about 503s on /health but the auto-scaling policy had cooldown = 300 seconds and the replacement node took 7 minutes to come up because the init script downloaded 1.2 GB of fonts for a demo dashboard. The payment service still didnt retry DNS, so the first 480 requests failed.
