We had a simple scheduled job: every hour, pull new products from a third party API. It worked fine on one server. Then we scaled to ten.
Suddenly, every hour, all ten servers fired the same job at the same time. Ten servers hitting the same third party API for the same data, ten times the work for a job that only needed to run once. It was pure waste: extra load, extra API calls, extra cost, for nothing. And in the worst case, if the job wasn't careful about how it wrote to the database, you could end up with duplicate rows or repeated updates piling up underneath all that wasted effort.
Here's how we fixed it.
The fix: only one server should win
The idea is simple. Every server's scheduler still fires on time, same as before. But before any of them actually do the work, they race to grab a lock. Whoever gets it runs the job. Everyone else just steps aside.
