Gunicorn worker timeout error

Question

I have setup gunicorn with 3 workers 30 worker connections and using eventlet worker class. It is setup behind Nginx. After every few requests, I see this in the logs.

Answer 1

You need to used an other worker type class an async one like gevent or tornado see this for more explanation : First explantion :

You may also want to install Eventlet or Gevent if you expect that your application code may need to pause for extended periods of time during request processing

Second one :

The default synchronous workers assume that your application is resource bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. For instance, a request to the internet meets this criteria. At some point the external network will fail in such a way that clients will pile up on your servers.

Answer 2

On Google Cloud Just add --timeout 90 to entrypoint in app.yaml

entrypoint: gunicorn -b :$PORT main:app --timeout 90

Answer 3

Is this endpoint taking too many time?

Maybe you are using flask without assynchronous support, so every request will block the call. To create async support without make difficult, add the gevent worker.

With gevent, a new call will spawn a new thread, and you app will be able to receive more requests

pip install gevent
gunicon .... --worker-class gevent

Answer 4

I've got the same problem in Docker.

In Docker I keep trained LightGBM model + Flask serving requests. As HTTP server I used gunicorn 19.9.0. When I run my code locally on my Mac laptop everything worked just perfect, but when I ran the app in Docker my POST JSON requests were freezing for some time, then gunicorn worker had been failing with [CRITICAL] WORKER TIMEOUT exception.

I tried tons of different approaches, but the only one solved my issue was adding worker_class=gthread.

Here is my complete config:

import multiprocessing

workers = multiprocessing.cpu_count() * 2 + 1
accesslog = "-" # STDOUT
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(q)s" "%(D)s"'
bind = "0.0.0.0:5000"
keepalive = 120
timeout = 120
worker_class = "gthread"
threads = 3

Answer 5

timeout is a key parameter to this problem.

however it's not suit for me.

i found there is not gunicorn timeout error when i set workers=1.

when i look though my code, i found some socket connect (socket.send & socket.recv) in server init.

socket.recv will block my code and that's why it always timeout when workers>1

hope to give some ideas to the people who have some problem with me

Answer 6

For me, the solution was to add --timeout 90 to my entrypoint, but it wasn't working because I had TWO entrypoints defined, one in app.yaml, and another in my Dockerfile. I deleted the unused entrypoint and added --timeout 90 in the other.