How can I run long tasks on Google App Engine, which uses gunicorn?

Question

GAE flex uses gunicorn as an entrypoint by default which is fine, except I have a function that takes a very long time to process (scraping websites and story data in a db)

Answer 1

You can use async worker-class and then you won't need to set the timeout to 20 minutes. The default worker class is sync. Docs regarding the workers here.

Use the eventlet async worker (gevent not recommended if using google client libraries)

pip install eventlet

Then in your gunicorn instantiation set the worker-class = 'eventlet' and set number of workers to [number of cores] x 2 +1 (that's just a recommendation in google docs). For example:

CMD exec gunicorn --worker-class eventlet --workers 3 -b :$PORT main:app

Gunicorn Worker Configuration

Alternatively, use implementation described here using pubsub and workers.