Nginx upstream prematurely closed connection while reading response header from upstream, for large requests

Question

I am using nginx and node server to serve update requests. I get a gateway timeout when I request an update on large data. I saw this error from the nginx error logs :

<

Answer 1

I solved this by setting a higher timeout value for the proxy:

location / {
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;
    proxy_pass http://localhost:3000;
}

Documentation: https://nginx.org/en/docs/http/ngx_http_proxy_module.html

Answer 2

I ran into this issue as well and found this post. Ultimately none of these answers solved my problem, instead I had to put in a rewrite rule to strip out the location /rt as the backend my developers made was not expecting any additional paths:

┌─(william@wkstn18)──(Thu, 05 Nov 20)─┐
└─(~)──(16:13)─>wscat -c ws://WebsocketServerHostname/rt
error: Unexpected server response: 502

Testing with wscat repeatedly gave a 502 response. Nginx error logs provided the same upstream error as above, but notice the upstream string shows the GET Request is attempting to access localhost:12775/rt and not localhost:12775:

 2020/11/05 22:13:32 [error] 10175#10175: *7 upstream prematurely closed
 connection while reading response header from upstream, client: WANIP,
 server: WebsocketServerHostname, request: "GET /rt/socket.io/?transport=websocket
 HTTP/1.1", upstream: "http://127.0.0.1:12775/rt/socket.io/?transport=websocket",
 host: "WebsocketServerHostname"

Since the devs had not coded their websocket (listening on 12775) to expect /rt/socket.io but instead just /socket.io/ (NOTE: /socket.io/ appears to just be a way to specify websocket transport discussed here). Because of this, rather than ask them to rewrite their socket code I just put in a rewrite rule to translate WebsocketServerHostname/rt to WebsocketServerHostname:12775 as below:

upstream websocket-rt {
        ip_hash;

        server 127.0.0.1:12775;
}

server {
        listen 80;
        server_name     WebsocketServerHostname;

        location /rt {
                proxy_http_version 1.1;

                #rewrite /rt/ out of all requests and proxy_pass to 12775
                rewrite /rt/(.*) /$1  break;

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $host;

                proxy_pass http://websocket-rt;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }

}

Answer 3

I had the same error for quite a while, and here what fixed it for me.

I simply declared in service that i use what follows:

Description= Your node service description
After=network.target

[Service]
Type=forking
PIDFile=/tmp/node_pid_name.pid
Restart=on-failure
KillSignal=SIGQUIT
WorkingDirectory=/path/to/node/app/root/directory
ExecStart=/path/to/node /path/to/server.js

[Install]
WantedBy=multi-user.target

What should catch your attention here is "After=network.target". I spent days and days looking for fixes on nginx side, while the problem was just that. To be sure, stop running the node service you have, launch the ExecStart command directly and try to reproduce the bug. If it doesn't pop, it just means that your service has a problem. At least this is how i found my answer.

For everybody else, good luck!