Specifying retry limit for tasks queued using GAE deferred library

Question

We are offloading certain time consuming tasks using the GAE deferred library and would like to know how can we set the retry limit for those offloaded tasks. We are running into issues where certain tasks are retried for ever as the task would never succeed due to some un recoverable exception.

Answer 1

According to the documentation the _retry_options of the deferred.defer API can be used to pass retry options to the associated Task() instance:

_countdown, _eta, _headers, _name, _target, _transactional, _url, _retry_options, _queue: Passed through to the task queue - see the task queue documentation for details.

From the Task() doc:

...

And you can use the TaskRetryOptions()'s task_retry_limit property:

task_retry_limit

The maximum number of retry attempts for a failed task.

In push queues, the counter is incremented each time App Engine tries the task, up to the specified task_retry_limit. If specified with task_age_limit, App Engine retries the task until both limits are reached.

In pull queues, the counter is incremented each time the task is leased, up to the specified task_retry_limit. Tasks are deleted automatically once they have been leased the number of times specified in the limit.

Note: the answer is based on documentation only, I didn't actually implemented it, YMMV.

Answer 2

According to the documentation

queue:
- name: fooqueue
  rate: 1/s
  retry_parameters:
    task_retry_limit: 7
    task_age_limit: 2d
- name: barqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 0
- name: bazqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 3

Answer 3

Check X-Appengine-Taskretrycount and X-Appengine-Taskexecutioncount http header values in your task.

If you don't want to retry a task, you can raise deferred.PermanentTaskFailure exception. This exception will be logged only, task won't run again.

Different ways to access http headers:

For http handlers triggered by taskqueue: num_tries = self.request.headers.get('X-AppEngine-TaskRetryCount')

For functions triggered by deferred library: num_tries = webapp2.get_request().headers.get('X-AppEngine-TaskRetryCount')