Idempotency, vendor failure, monitoring that catches the invisible outages, and the tradeoffs nobody warns you about, lessons from scaling payments infrastructure.
Most software fails quietly. A page renders slowly, a recommendation is a little off, a report is stale by an hour. Users shrug and move on.
Payments doesn't work like that. When payments break, someone's money is in a place neither of you can account for, and the clock starts ticking on their patience. There's no graceful degradation. Either the money moved, or it didn't, and someone needs to know which.
I've spent a good chunk of my career building and scaling payments infrastructure, and it has quietly rewired how I think about engineering in general. Here's what stuck.
#
