How to run long cron jobs on App Engine flexible environment?

Question

I have an app on App Engine (flexible environment) and configured a few cron jobs. These jobs should take several minutes but I see them failing after ~30 seconds (502 error). The documentation is not very clear regarding the max time of cron jobs (Scheduling Jobs with cron.yaml), although it seems that "An HTTP request invoked by cron can run for up to 24 hours".

Any ideas of how to overcome this? Thanks in advance

Answer 1

This is an answer to my own question.

The problem I had was that I only had one Gunicorn worker. The App Engine health checks were happening every 30 seconds and there was no worker able to reply to the health checks, so the server was restarted.

I should have added more workers in the app.yaml file. For example, I've added the following line.

entrypoint: gunicorn -b :$PORT main:app --workers 12

Hope this helps.