The problem nobody talks about
You have a payment gateway. It fails sometimes. So you add a retry.
Now you have a worse problem: a customer clicks "Pay", the request reaches Stripe, the charge goes through, but the response never comes back. Your retry fires. Stripe charges them again.
That's not a hypothetical. It's the default behavior of any naive retry implementation, and it happens in production every day.
This post is about how we built a checkout system that handles this correctly — with retry that never double-charges, a circuit breaker that protects the service when the gateway is degraded, and a feedback loop that adjusts its own configuration under load.
