How to get the failed tasks in Celery?

Question

I am using celery to process some tasks. I can see how many are active or scheduled etc, but I am not able to find any way to see the tasks that have failed. Flower does sho

Answer 1

Celery doesn't make it easy to find a failed task but Flower (the main Celery management web app) does simplify this. It keeps a record of task IDs even after they are completed, and has an API to let you find only failed tasks.

Flower's rather basic HTTP API includes the /api/tasks endpoint - you can use /api/tasks?state=FAILURE to show only failed tasks, then parse the JSON to extract what you need. The contents is similar to what you get in the web API, and it's easy to prototype with curl and format/filter with jq:

curl -s 'http://localhost:5555/api/tasks?state=FAILURE&limit=5' | jq . | less

Flower needs to be installed and running of course.

Since you have millions of completed tasks, you may need to capture failed task info in a data store for efficient access - perhaps Flower will help. Or you could try a custom on-failure handler in Celery, to capture just failed task info - see this answer.