Retries keep distributed systems moving through timeouts and temporary failures. In an access system, though, an old retry can be more dangerous than a failed request.
This is Part 2 of Security Infrastructure in Practice, a series about what happens when security design meets production systems.
Access had been removed from a deactivated account. The audit trail showed a successful removal. A few minutes later, the membership was back !!!
No administrator had restored it. An older provisioning job had timed out, waited, and replayed the grant after the deactivation.
The sequence looked like this:
