The first time we deployed Celery to production on a client project, we thought we had done everything right. We had workers running, tasks queuing, and Redis as the broker. Six weeks later, the task queue was backed up with 40,000 unprocessed jobs, the workers had silently died, nobody knew, and a batch of client invoices had not been generated for two weeks.
That was four years ago. Since then we have deployed Celery on dozens of projects and we have learned what actually goes wrong — not in development, where everything works, but in production, where things fail in ways you do not anticipate.
This post covers the configuration and operational patterns we now use on every Celery deployment.
Why tasks fail silently (and how to stop it)
The most dangerous thing about Celery is how quietly it can fail. A worker process dies, the task queue fills up, and your application keeps accepting work and sending it to a queue that nobody is processing. No exception is raised. No alert fires. Users notice eventually, or you notice when a daily report does not arrive.
