Celery: Worker with concurrency and reserved tasks only running 1 task

Question

Some of the tasks in my code were taking longer and longer to execute.

Upon inspection I noticed that although I have my worker node set to concurrency 6, and 6 processe

Answer 1

The docs mention here how to reserve one task at a time - or only as many as you have concurrency:

Often users ask if disabling “prefetching of tasks” is possible, but what they really mean by that, is to have a worker only reserve as many tasks as there are worker processes (10 unacknowledged tasks for -c 10)

That’s possible, but not without also enabling late acknowledgment. Using this option over the default behavior means a task that’s already started executing will be retried in the event of a power failure or the worker instance being killed abruptly, so this also means the task must be idempotent ... You can enable this behavior by using the following configuration options:

task_acks_late = True
worker_prefetch_multiplier = 1

or the code equivalent:

app = Celery(...)
app.conf.worker_prefetch_multiplier = 1
app.conf.task_acks_late = True
...