How do I prevent Socket/Port Exhaustion?

Question

I am attempting to performance test a website by hitting it with requests across multiple threads. Each thread executes n times. (in a for loop)

However, I

Answer 1

You don't need to mess around with TIME_WAIT to accomplish what you want.

The problem is that you are disposing the WebClient every time you call Execute(). When you do that, you close the socket connection with the server and the TCP port keeps busy for the TIME_WAIT period.

A better approach is to create the WebClient in the constructor of your HttpGetTest class and reuse the same object throughout the test.

WebClient uses keep alive by default and will reuse the same connection for all its requests so in your case there will be only 100 opened connections for this.

Answer 2

It looks like you are not forcing your WebClient to get rid of the resources that it has allocated. You are performing a Using on the stream that is returned, but your WebClient still has resources.

Either wrap your WebClient instantiation in a using block, or manually call dispose on it once you are done reading from the URL.

Try this:

public sealed class HttpGetTest : ITest {
    private readonly string m_url;

    public HttpGetTest( string url ) {
        m_url = url;        
    }

    public void ITest.Execute() {
        using( var m_webClient = new WebClient())
        {
            using( Stream stream = m_webClient.OpenRead( m_url ) ) 
            {

            }
        }
    }
}

Answer 3

Q: How do I explicitly close a socket ... in order to prevent TIME_WAIT states?

A: Dude, TIME_WAIT is an integral - and important! - part of TCP/IP itself!

You can tune the OS to reduce TIME_WAIT (which can have negative repercussions).

And you can tune the OS to increase #/ephemeral ports:

• http://msdn.microsoft.com/en-us/library/aa560610%28v=bts.20%29.aspx

Here's a link on why TIME_WAIT exists ... and why it's a Good Thing:

• http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html

Answer 4

It's not an issue of closing sockets or releasing resources in your app. The TIME _WAIT is a TCP stack timeot on released sockets to prevent their re-use until such time as it is virtually impossible for any packets 'left over' from a previous connection to that socket to not have expired.

For test purposes, you can reduce the wait time from the default, (some minutes, AFAIK), to a smaller value. When load-testing servers, I set it at six seconds.

It's in the registry somewhere - you'll find it if you Google.

Found it:

Change TIME_WAIT delay

Answer 5

Do you understand the purpose of TIME_WAIT? It's a period during which it would be unsafe to reuse the port because lost packets (that have been successfully retransmitted) from the previous transaction might yet be delivered within that time period.

You could probably tweak it down in the registry somewhere, but I question if this is a sensible next step.

My experience of creating realistic load in a test environment have proved very frustrating. Certainly running your load-tester from localhost is by no means realistic, and most network tests I have made using the .net http apis seem to require more grunt in the client than the server itself.

As such, it's better to move to a second machine for generating load on your server... however domestic routing equipment is rarely up to the job of supporting anywhere near the number of connections that would cause any sort of load on a well written server app, so now you need to upgrade your routing/switching equipment as well!

Lastly, I've had some really strange and unexpected performance issues around the .net Http client API. At the end of the day, they all use HttpWebRequest to do the heavy lifting. IMO it's nowhere near as performant as it could be. DNS is sychronous, even when calling the APIs asynchronously (although if you're only requesting from a single host, this isn't an issue), and after sustained usage CPU usage creeps up until the client becomes CPU constrained rather than IO constrained. If you're looking to generate sustained and heavy load, any request-heavy app reliant on HttpWebRequest is IMO a bogus investment.

All in all, a pretty tricky job, and ultimately, something that can only be proved in the wild, unless you've got plently of cash to spend on an armada of better equipment.

[Hint: I got much better perfomance from my own client written using async Socket apis and a 3rd party DNS client library]