Why does a simple Thin server stop responding at 16500 requests when benchmarking? [duplicate]

Answer 1

I'll add the solution here for claritys sake. The correct solution for managing to do high frequency tests with ab on os X is to change the 'net.inet.tcp.msl' setting from 15000ms to 1000ms. This should only be done on development boxes.

 sudo sysctl -w net.inet.tcp.msl=1000 # this is only good for local development

This answer was found after the good detective work performed in the comments here and comes from an answer to a very similar question here's the answer: https://stackoverflow.com/a/6699135/155031

Answer 2

I think I've got it.

When ab makes connections to your test server, it opens a source port (say, 50134) and makes a connection to the destination port (9294).

The ports that ab opens for the source port are determined by the sysctl settings net.inet.ip.portrange.first and net.inet.ip.portrange.last. For example, on my machine:

philippotter ~ $ sysctl -a | grep ip.portrange
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535

This means that ab's source ports will be in the range from 49152 to 65535, which is a total of 16384.

HTTP is a TCP protocol. When a TCP connection is closed, it goes into the TIME_WAIT state, while it waits for any remaining in-transit packets to reach their destinations. This means that the port is not usable for any other purpose until the timeout is reached.

So, putting all of this together, ab uses up all available source ports very quickly; they go into the TIME_WAIT state; they can't be reused; ab is unable to create any more connections.

You can see this if you kill ab when it hangs, and run it again -- it won't be able to create any connections!