celery shutdown worker after particular task

Question

I\'m using celery (solo pool with concurrency=1) and I want to be able to shut down the worker after a particular task has run. A caveat is that I want to avoid any possibility

Answer 1

If you shutdown the worker, after the task has completed, it won't re-queue again.

@task_postrun.connect(sender=add)
def shutdown(*args, **kwargs):
    app.control.broadcast('shutdown')

This will gracefully shutdown the worker after tasks is completed.

[2018-04-01 18:44:14,627: INFO/MainProcess] Connected to redis://localhost:6379/0
[2018-04-01 18:44:14,656: INFO/MainProcess] mingle: searching for neighbors
[2018-04-01 18:44:15,719: INFO/MainProcess] mingle: all alone
[2018-04-01 18:44:15,742: INFO/MainProcess] celery@foo ready.
[2018-04-01 18:46:28,572: INFO/MainProcess] Received task: celery_worker_stop.add[ac8a65ff-5aad-41a6-a2d6-a659d021fb9b]
[2018-04-01 18:46:28,585: INFO/ForkPoolWorker-4] Task celery_worker_stop.add[ac8a65ff-5aad-41a6-a2d6-a659d021fb9b] succeeded in 0.005628278013318777s: 3   
[2018-04-01 18:46:28,665: WARNING/MainProcess] Got shutdown from remote

Note: broadcast will shutdown all workers. If you want to shutdonw a specific worker, start worker with a name

celery -A celeryapp  worker -n self_killing --concurrency=1 --pool=solo

Now you can shutdown this with destination parameter.

app.control.broadcast('shutdown', destination=['celery@self_killing'])

Answer 2

The recommended process for shutting down a worker is to send the TERM signal. This will cause a celery worker to shutdown after completing any currently running tasks. If you send a QUIT signal to the worker's main process, the worker will shutdown immediately.

The celery docs, however, usually discuss this in terms of managing celery from a command line or via systemd/initd, but celery additionally provides a remote worker control API via celery.app.control.
You can revoke a task to prevent workers from executing the task. This should prevent the loop you are experiencing. Further, control supports shutdown of a worker in this manner as well.

So I imagine the following will get you the behavior you desire.

@app.task(bind=True)
def shutdown(self):
    app.control.revoke(self.id) # prevent this task from being executed again
    app.control.shutdown() # send shutdown signal to all workers

Since it's not currently possible to ack the task from within the task, then continue executing said task, this method of using revoke circumvents this problem so that, even if the task is queued again, the new worker will simply ignore it.

Alternatively, the following would also prevent a redelivered task from being executed a second time...

@app.task(bind=True)
def some_task(self):
    if self.request.delivery_info['redelivered']:
        raise Ignore() # ignore if this task was redelivered
    print('This should only execute on first receipt of task')

Also worth noting AsyncResult also has a revoke method that calls self.app.control.revoke for you.

Answer 3

If you need to shutdown a specific worker and don't know it's name in advance, you can get it from the task properties. Based on the answers above, you can use:

app.control.shutdown(destination=[self.request.hostname])

or

app.control.broadcast('shutdown', destination=[self.request.hostname])

Note:

• A worker should be started with a name (option '-n');
• The task should be defined with bind=True parameter.