Prioritizing queues among multiple queues in celery?

Question

We are using celery for our asynchronous background tasks and we have 2 queues for different priority tasks. We have 2 cluster of nodes serving them separately. Things are worki

Answer 1

You can partially achieve this by defining multiple queues for the worker, when starting it.

You can do it with the following command: Also, refer here for more details.

celery -A proj worker -l info -Q Q1,Q2

Though this approach has a problem. It doesn't do it with fallback kind of approach. Since, workers listening to multiple queue evenly distribute the resources among them.

Hence, your requirement of processing only from 'high priority queue' even when there is something in 'normal priority queue' cannot be achieved. This can be minimized by allocating more Workers (may be 75%) for 'high priority queue' and 25% for 'normal priority queue'. or different share based on you work load.