celerybeat - multiple instances & monitoring

Question

I\'m having application built using celery and recently we got a requirement to run certain tasks on schedule.

I think celerybeat is perfect for this, but I got few ques

Answer 1

I just found this solution as celery-beat replacement: RedBeat, blog post

Didn't use it yet though.

Answer 2

To answer your 2 questions:

1. If you run several celerybeat instances you get duplicated tasks, so afaik you should have only single celerybeat instance.

2. I'm using supervisord as you mentioned to run celery workers and celerybeat workers as deamon so they should always be up & running.

my supervisord config:

[program:my_regular_worker]
command=python2.7 /home/ubuntu/workspace/src/manage.py celery worker -Q my_regular_worker-queue_name -c 1 -l info --without-mingle
process_name=my_regular_worker
directory=/home/ubuntu/workspace/src
autostart=true
autorestart=true
user=ubuntu
stdout_logfile=/tmp/my_regular_worker.log
redirect_stderr=true



[program:my_celerybeat_worker]
command=python2.7 /home/ubuntu/workspace/src/manage.py celery worker -Q my_celerybeat_worker-queue_name -c 1 -l info --without-mingle -B -s /tmp/celerybeat-schedule

Answer 3

You may run multiple instances of celery beat and tasks will not be duplicated.

Take a look at the celery.beat.Scheduler class, specifically the reserve() function. The scheduler will reserve a task before submitting it to the grid for execution. This prevents another instance of celery beat from submitting the same task.

We use MongoDB as a backing store for our scheduled tasks. Here is a sample document showing that the task has been reserved by one of the schedulers.

{
  "startdate": "2015-07-06 00:00:00", 
  "task": "cobalt.grid.tasks_facts.task_add", 
  "enddate": "2018-01-01 00:00:00", 
  "args": "[13.0, 42.0]", 
  "enabled": "True", 
  "last_run_at": "2015-08-13 15:04:49.058000", 
  "interval": "{u'every': u'1', u'period': u'minutes'}", 
  "relative": "False", 
  "total_run_count": "12", 
  "kwargs": "{}", 
  "reserved": "compute2:25703", 
  "_id": "ObjectId(55ccaf7784a3e752e73b08c2)", 
  "options": "{}"
}

http://celery.readthedocs.org/en/latest/reference/celery.beat.html#celery.beat.Scheduler