You want to run a job every night at 2am. Easy, cron does that. Now you want to run a million jobs, each on its own schedule, across a fleet of machines where any node can die mid-execution, and you want each job to fire on time and not twice. Cron on one box no longer cuts it. This is the problem a distributed job scheduler solves, and the interesting parts are all about failure.
The core problem
A scheduler has two responsibilities that are easy to confuse: deciding when a job should run, and actually running it. The first is a time problem. The second is an execution problem. Bundling them into one process is what makes single-node cron simple and also what makes it fragile. Once you distribute, you separate them.
The two hard guarantees are: a job that is due must eventually run (durability), and a job should not run twice when you did not ask it to (correctness). Most naive designs quietly violate one of these the first time a worker crashes.
Key design decisions
