Pros and cons to use Celery vs. RQ [closed]

Question

Currently I'm working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience with these job queues, but I want to ask you guys to share you experience of using this tools. So.

1. What pros and cons to use Celery vs. RQ.
2. Any examples of projects/task suitable to use Celery vs. RQ.

Celery looks pretty complicated but it's full featured solution. Actually I don't think that I need all these features. From other side RQ is very simple (e.g configuration, integration), but it seems that it lacks some useful features (e.g task revoking, code auto-reloading)

Answer 1

Here is what I have found while trying to answer this exact same question. It's probably not comprehensive, and may even be inaccurate on some points.

In short, RQ is designed to be simpler all around. Celery is designed to be more robust. They are both excellent.

• Documentation. RQ's documentation is comprehensive without being complex, and mirrors the project's overall simplicity - you never feel lost or confused. Celery's documentation is also comprehensive, but expect to be re-visiting it quite a lot when you're first setting things up as there are too many options to internalize

• Monitoring. Celery's Flower and the RQ dashboard are both very simple to setup and give you at least 90% of all information you would ever want

• Broker support. Celery is the clear winner, RQ only supports Redis. This means less documentation on "what is a broker", but also means you cannot switch brokers in the future if Redis no longer works for you. For example, Instagram considered both Redis and RabbitMQ with Celery. This is important because different brokers have different guarantees e.g. Redis cannot (as of writing) guarantee 100% that your messages are delivered.

• Priority queues. RQs priority queue model is simple and effective - workers read from queues in order. Celery requires spinning up multiple workers to consume from different queues. Both approaches work

• OS Support. Celery is the clear winner here, as RQ only runs on systems that support fork e.g. Unix systems

• Language support. RQ only supports Python, whereas Celery lets you send tasks from one language to a different language

• API. Celery is extremely flexible (multiple result backends, nice config format, workflow canvas support) but naturally this power can be confusing. By contrast, the RQ api is simple.

• Subtask support. Celery supports subtasks (e.g. creating new tasks from within existing tasks). I don't know if RQ does

• Community and Stability. Celery is probably more established, but they are both active projects. As of writing, Celery has ~3500 stars on Github while RQ has ~2000 and both projects show active development

In my opinion, Celery is not as complex as its reputation might lead you to believe, but you will have to RTFM.

So, why would anyone be willing to trade the (arguably more full-featured) Celery for RQ? In my mind, it all comes down to the simplicity. By restricting itself to Redis+Unix, RQ provides simpler documentation, simpler codebase, and a simpler API. This means you (and potential contributors to your project) can focus on the code you care about, instead of having to keep details about the task queue system in your working memory. We all have a limit on how many details can be in our head at once, and by removing the need to keep task queue details in there RQ lets get back to the code you care about. That simplicity comes at the expense of features like inter-language task queues, wide OS support, 100% reliable message guarantees, and ability to switch message brokers easily.

Answer 2

Celery is not that complicated. At its core, you do the step by step configuration from the tutorials, create a celery instance, decorate your function with @celery.task then run the task with my_task.delay(*args, **kwargs).

Judging from your own assessment, it seems you have to choose between lacking (key) features or having some excess hanging around. That is not too hard of a choice in my book.