A payment request goes out. The network hiccups. The client never sees a response, so it does the reasonable thing and retries. Thirty seconds later, a customer gets charged twice for one order, and someone on support is trying to explain to them why.
Nobody wrote a bug here. The client behaved exactly the way retry logic is supposed to behave. The problem is one layer deeper: the system let a "did this actually happen" question get answered twice, and each answer was "yes, charge them."
That's idempotency, or rather, the absence of it. It's one of those concepts that's simple to state and easy to get wrong in practice, and once you've been burned by it once, you start seeing the gap everywhere.
Why retries are unavoidable, not optional
In a distributed system, a timeout doesn't tell you the request failed. It tells you that you didn't get a response in time. The request might have never arrived. It might have arrived, succeeded, and the response got lost on the way back. From the caller's side, those two situations are indistinguishable, and the only reasonable move is to retry.
