Your client sends a POST /products. The server creates the product, then the load balancer kills the connection before the response gets out. The client sees a timeout. What does every reasonable HTTP client do on a timeout? It retries. Now you have two products.
This is not an exotic failure mode. It's Tuesday. Mobile clients on flaky networks, Kubernetes rolling deploys, aggressive proxy timeouts — anything that separates "the server did the work" from "the client learned about it" will eventually duplicate your writes. The standard fix is an idempotency key: the client generates a unique key per logical operation, sends it with the request, and the server guarantees that the same key never executes the operation twice.
The concept is simple. The implementation is where I got burned, so let me walk through the version I actually ship now, including the parts I got wrong the first time.
The naive version (which I shipped)
Here's the implementation almost everyone writes first, me included:
