Celery beat - different time zone per task

Question

I am using celery beat to schedule some tasks. I'm able to use the CELERY_TIMEZONE setting to schedule the tasks using the crontab schedule and it runs at the scheduled time in the mentioned time zone.

But I want to be able to setup multiple such tasks for different timezones in the same application (single django settings.py). I know which task needs to run in what timezone when the task is being scheduled.

Is it possible to specify a different timezone for each of the tasks?

I'm using django (1.4) with celery (3.0.11) and django celery (3.0.11).

I've looked at the djcelery.schedulers.DatabaseScheduler class and it's base class, but I can't figure out how and where the timezone is getting used. Can I write a custom scheduler that can make each job run in a different timezone?

Thanks,

Answer 1

You can achieve a timezone-aware scheduling of individual tasks in a celery schedule. This way you can run a task according to the local time in a specific timezone (also adjusting to e.g. daylight saving time) by specifying a separate now function for each celery schedule

crontab supports the nowfun argument to specify the datetime function to be used to check if it should run

import datetime
import pytz
nowfun = lambda: datetime.datetime.now(pytz.timezone('Europe/Berlin'))

In your schedule, set this function as the datetime function via

'periodic_task': {
    'task': 'api.tasks.periodic',
    'schedule': crontab(hour=6, minute=30, nowfun=nowfun)
}

This runs every day at 6.30am CET adjusted to daylight savings.

In case you use the function more than once, consider creating a helper

from functools import partial
cet_crontab = partial(crontab, nowfun=nowfun)
'periodic_task': {
    'task': 'api.tasks.periodic',
    'schedule': cet_crontab(hour=6, minute=30)
}

Make sure you have CELERY_ENABLE_UTC = False set, otherwise celery converts your schedules to UTC.

Answer 2

I think the easiest way to do it is to use decorator with mock

from mock import patch

@task
@patch.multiple(settings, CELERY_TIMEZONE='time_zone')
def my_task(*args):
    #do your staff

I'm not tested it yet but it seems right. I hope i helped you :)

Answer 3

Maybe I'm misunderstanding the problem, but if you want to run tasks at a certain time in different time zones, could you not just offset the scheduled time by the hour difference between the time zones? For example, if I wanted a task to run at 5 PM in two different TZs:

# settings.py
CELERY_TIMEZONE = "US/Eastern"

CELERYBEAT_SCHEDULE = { 
    # Task to run in EST
    'schedule-my-est-task': {
        'task': 'path.to.my.est.task',
        'schedule': crontab(minute=0, hour=17),
    },
    # Task to run in UTC - hour time is offset
    'schedule-my-utc-task': {
        'task': 'path.to.my.utc.task',
        'schedule': crontab(minute=0, hour=10), 
    },
}