Week one of the pilot, our voice agent booked appointments for a dental group. Forty operatories, three locations, one phone line per office that never stopped ringing. The agent took the call, checked the calendar, wrote the slot, read it back. Clean.
The 9pm page came on a Thursday. Front desk had found four double-booked slots from that afternoon. Same callers, same times, two rows each in the scheduling table. The agent swore (in the transcript) it booked once. The database swore it booked twice. Both were telling the truth.
Here is what actually happened. Our booking call went out to the practice management API. That API was slow that day, p95 around 4200ms, sometimes worse. We had a 3000ms timeout on the HTTP client. So the request would land, the booking would commit on their side, and our client would give up waiting before the 201 came back. The agent saw a timeout, treated it as a failure, and said the line every voice agent says when something goes wrong: "sorry, let me try that again." Then it fired the same booking a second time. The second one was fast enough to return. Two rows. One confused Mrs. Alvarez.
The retry was the bug. Not the slowness. Slowness is normal. The sin was retrying a write that had no idempotency key, so the downstream system had no way to know the second request was the same intent as the first.
