On December 16, 2024, a developer filed a bug report against Trigger.dev, the open-source background job framework. A routine nightly server restart had caused random tasks to get stuck in a "queued" state. The system's recovery logic, working exactly as designed, detected the stalled tasks and requeued them. Then it detected them again. And again. By the time anyone checked the dashboard, 3,800 duplicate tasks were sitting in the queue, each one a faithful copy of work that had already been completed.
The monitoring system showed no errors. Every task had succeeded. The duplicates were executing successfully too. From the system's perspective, nothing was wrong.
This is the kind of bug that makes senior engineers go quiet. Not because it's complicated — the explanation fits in a sentence — but because the implications are uncomfortable. The system didn't malfunction. It did exactly what it was designed to do: detect abandoned work and retry it. The problem is that "completed successfully" and "abandoned silently" produce the same signal from the outside. Both go quiet.
Two Generals and No Good Options
The theoretical foundation for this is older than most production systems running today. In 1985, Fischer, Lynch, and Paterson published their impossibility result in the Journal of the ACM: given the possibility of even a single faulty process, it is impossible for a system of processes to agree on a decision. The paper is formally about consensus, but its practical implication is about something more mundane. It's about acknowledgments.
