How to restart Celery gracefully without delaying tasks

Question

We use Celery with our Django webapp to manage offline tasks; some of these tasks can run up to 120 seconds.

Whenever we make any code modifications, we need to rest

Answer 1

I've recently fixed the bug with SIGHUP: https://github.com/celery/celery/pull/662

Answer 2

Can you launch it with a custom pid file name. Possibly timestamped, and key off of that to know which PID to kill?

CELERYD_PID_FILE="/var/run/celery/%n_{timestamp}.pid"

^I dont know the timestamp syntax but maybe you do or you can find it?

then use the current system time to kill off any old pids and launch a new one?

Answer 3

rm *.pyc

This causes the updated tasks to be reloaded. I discovered this trick recently, I just hope there are no nasty side effects.

Answer 4

I think you can try this:

kill -s HUP ``cat /var/run/celeryd.pid`` 
python manage.py celeryd --pidfile=/var/run/celeryd.pid

HUP may recycle every free worker and leave executing workers keep running and HUP will let these workers be trusted. Then you can safely restart a new celery worker main process and workers. Old workers may be killed itself when task has been finished.

I've use this way in our production and it seems safe now. Hope this can help you!

Answer 5

Well you using SIGHUP (1) for warm shutdown of celery. I am not sure if it actually causes a warm shutdown. But SIGINT (2) would cause a warm shutdown. Try SIGINT in place of SIGHUP and then start celery manually in your script (I guess).

Answer 6

celeryd has --autoreload option. If enabled, celery worker (main process) will detect changes in celery modules and restart all worker processes. In contrast to SIGHUP signal, autoreload restarts each process independently when the current executing task finishes. It means while one worker process is restarting the remaining processes can execute tasks.

http://celery.readthedocs.org/en/latest/userguide/workers.html#autoreloading