Django run tasks (possibly) in the far future

问题

Suppose I have a model Event. I want to send a notification (email, push, whatever) to all invited users once the event has elapsed. Something along the lines of:

class Event(models.Model):
    start = models.DateTimeField(...)
    end = models.DateTimeField(...)
    invited = models.ManyToManyField(model=User)

    def onEventElapsed(self):
        for user in self.invited:
           my_notification_backend.sendMessage(target=user, message="Event has elapsed")

Now, of course, the crucial part is to invoke onEventElapsed whenever timezone.now() >= event.end. Keep in mind, end could be months away from the current date.

I have thought about two basic ways of doing this:

Use a periodic cron job (say, every five minutes or so) which checks if any events have elapsed within the last five minutes and executes my method.
Use celery and schedule onEventElapsed using the eta parameter to be run in the future (within the models save method).

Considering option 1, a potential solution could be django-celery-beat. However, it seems a bit odd to run a task at a fixed interval for sending notifications. In addition I came up with a (potential) issue that would (probably) result in a not-so elegant solution:

Check every five minutes for events that have elapsed in the previous five minutes? seems shaky, maybe some events are missed (or others get their notifications send twice?). Potential workaroung: add a boolean field to the model that is set to True once notifications have been sent.

Then again, option 2 also has its problems:

Manually take care of the situation when an event start/end datetime is moved. When using celery, one would have to store the taskID (easy, ofc) and revoke the task once the dates have changed and issue a new task. But I have read, that celery has (design-specific) problems when dealing with tasks that are run in the future: Open Issue on github. I realize how this happens and why it is everything but trivial to solve.

Now, I have come across some libraries which could potentially solve my problem:

celery_longterm_scheduler (But does this mean I cannot use celery as I would have before, because of the differend Scheduler class? This also ties into the possible usage of django-celery-beat... Using any of the two frameworks, is it still possible to queue jobs (that are just a bit longer-running but not months away?)
django-apscheduler, uses apscheduler. However, I was unable to find any information on how it would handle tasks that are run in the far future.

Is there a fundemantal flaw with the way I am approaching this? Im glad for any inputs you might have.

Notice: I know this is likely to be somehwat opinion based, however, maybe there is a very basic thing that I have missed, regardless of what could be considered by some as ugly or elegant.

回答1:

We're doing something like this in the company i work for, and the solution is quite simple.

Have a cron / celery beat that runs every hour to check if any notification needs to be sent. Then send those notifications and mark them as done. This way, even if your notification time is years ahead, it will still be sent. Using ETA is NOT the way to go for a very long wait time, your cache / amqp might loose the data.

You can reduce your interval depending on your needs, but do make sure they dont overlap.

If one hour is too huge of a time difference, then what you can do is, run a scheduler every hour. Logic would be something like

run a task (lets call this scheduler task) hourly that gets all notifications that needs to be sent in the next hour (via celery beat) -
Schedule those notifications via apply_async(eta) - this will be the actual sending

Using that methodology would get you both of best worlds (eta and beat)

来源：https://stackoverflow.com/questions/60439489/django-run-tasks-possibly-in-the-far-future

标签

python

django

celery

apscheduler

django-celery-beat