How to solve nginx - no live upstreams while connecting to upstream client?

Question

Currently I am running a load test using JMeter on our system build on grails 3 running on tomcat. After sending 20k request per second I got “no live upstreams while connec

Answer 1

For me, the issue was with my proxy_pass entry. I had

location / {
        ...
        proxy_pass    http://localhost:5001;
    }

This caused the upstream request to use the IP4 localhost IP or the IP6 localhost IP, but every now and again, it would use the localhost DNS without the port number resulting in the upstream error as seen in the logs below.

[27/Sep/2018:16:23:37 +0100]  - - -   to: [::1]:5001: GET /api/hc response_status 200
[27/Sep/2018:16:24:37 +0100]  - - -   to: 127.0.0.1:5001: GET /api/hc response_status 200
[27/Sep/2018:16:25:38 +0100]  - - -   to: localhost: GET /api/hc response_status 502
[27/Sep/2018:16:26:37 +0100]  - - -   to: 127.0.0.1:5001: GET /api/hc response_status 200
[27/Sep/2018:16:27:37 +0100]  - - -   to: [::1]:5001: GET /api/hc response_status 200

As you can see, I get a 502 status for "localhost:"

Changing my proxy_pass to 127.0.0.1:5001 means that all requests now use IP4 with a port.

This StackOverflow response was a big help in finding the issue as it detailed changing the log format to make it possible to see the issue.