A few years ago, I spent about a month cleaning up duplicate charges. Every service involved already had idempotency "handled": the client sends an Idempotency-Key header, we store the response under it, and if the key comes back we replay the stored response. Textbook. And it still double-charged people.
What finally clicked for me is that the key is just a name for the request. It doesn't do anything on its own. Whatever safety you have comes from the code around it, and that code is a small distributed state machine that's easy to get wrong. A few of the ways it went wrong for us:
Two requests with the same key arrive a millisecond apart. Both read "no record yet," both write one, both run. A check-then-act race. The fix is to make the write itself the check, one atomic insert, so exactly one wins.
A worker charges the card, then dies before it records "done." The key is stuck, and depending on how you wrote it, retries either hang forever or someone clears the key and you charge again. You need a lease that expires, and a way to reject the dead worker's late write (a fencing token). Otherwise crash recovery quietly turns into double execution.
Someone reuses one key for a $200 request and then a $500 one. If you key only on the token, you cheerfully hand back the $200 response. You have to fingerprint the request and reject the mismatch instead of replaying the wrong thing.
