node.js http.get hangs after 5 requests to remote site

Question

I\'m writing a simple api endpoint to determine if my server is able to reach the internet. It works great, but after 5 requests (exactly 5, every time) the request hangs. S

Answer 1

Here's the reason of the "exactly 5": https://nodejs.org/docs/v0.10.36/api/http.html#http_agent_maxsockets

Internally, the http module uses an agent class to manage HTTP requests. That agent will, by default, allow for a maximum of 5 open connections to the same HTTP server.

In your code, you don't consume the actual response sent by Google. So the agent assumes that you're not done with the request, and will keep the connection open. And so after 5 requests, the agent won't allow you to create a new connection anymore and will start waiting for any of the existing connections to complete.

The obvious solution would be to just consume the data:

http.get("http://www.google.com", function(r){
  r.on('data', function() { /* do nothing */ });
  ...
});

If you run into the problem that your /api/internetcheck route is called a lot, so you need to allow for more than 5 concurrent connections, you can either up the connection pool size, or just disable the agent completely (although you would still need to consume the data in both cases);

// increase pool size
http.globalAgent.maxSockets = 100;

// disable agent
http.get({ hostname : 'www.google.com', path : '/', agent : false }, ...)

Or perhaps use a HEAD request instead of GET.

(PS: in case http.get generates an error, you should still end the HTTP response by using res.end() or something like that).

NOTE: in Node.js versions >= 0.11, maxSockets is set to Infinity.

Answer 2

If you wait long enough, the 5 requests will time-out and the next 5 will process so the application isn't really hanging, because it will eventually process through all of the requests.

To speed up the process you need to do something with the response data such as r.on('data', function() {});