Increase celery retry time each retry cycle

Question

I do retries with celery like in the Docs-Example:

@task()
def add(x, y):
    try:
        ...
    except Exception, exc:
        add.retry(exc=exc, countdown=60)  # override the default and
                                          # retry in 1 minute

How can I increase the retry-countdown everytime the retry occurs for this job - e.g. 60 seconds, 2 minutes, 4 minutes and so on until the MaxRetriesExceeded is raised?

Answer 1

Since version 4.2 you can use options autoretry_for and retry_backoff for this purposes, for example:

@task(max_retries=10, autoretry_for=(Exception,), retry_backoff=60)
def add(x, y):
    pass

Answer 2

Here is a simple way to create bigger delay each time the task is evaluated. This value is updated by celery itself so you don't need to manage anything yourself.

@task()
def add(x, y):
    try:
        ...
    except Exception as exc:
        raise add.retry(exc=exc, countdown=60 * add.request.retries) 

Note: First task is repeated with countdown of 0. Because number of retries is 0 for the first run.

Answer 3

Keep a variable with your last retry time in it, and multiply it by 2 each time until it exceeds whatever level you want (or, keep a count if you prefer a certain number of times...)