.NET HttpWebRequest Speed versus Browser

Question

I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).

If I use HttpWebRequest to request an html page

Answer 1

Run the application with Ctrl+F5 instead of F5 (Debug mode). You will see a difference:

class Program
{
    static void Main()
    {
        using (var client = new WebClient())
        {
            Stopwatch watch = Stopwatch.StartNew();
            var data = client.DownloadData("http://news.bbc.co.uk");
            watch.Start();
            Console.WriteLine("{0} ms", watch.ElapsedMilliseconds);
        }
    }
}

Prints 880 ms on my PC.

Answer 2

The first time you request a page, .net tries to detect proxy settings. The solution is to pass in an empty WebProxy object. This way it just connects to remote server instead of autodetecting the proxy server.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriTest);
request.Proxy = new WebProxy();

Answer 3

If you make two requests does the second one happen more quickly?

I have also notice speed disparities between browsers and WebClient or WebRequest. Even the raw speed of the response can be drastically different - but not all the time!

There are a few things this could be caused by:

• It could be all the .Net bootstrapping that happens. .Net assemblies aren't loaded and JITted until they are used, therefore you can see significant speed degradation on the initial call to a piece of code even if the application itself has been running for ages. Okay - so the .Net framework itself is nGen'd - but there's still the bridge between your code and the .Net framework to build on the fly.

• Just checking that you're running without the debugger attached and that you definitely don't have symbol server switched on - symbol server and VS interrupts programs as the symbols are downloaded, slowing them down bucket-loads. Sorry if this is an insult ;)

• Browsers are coded to make efficient use of only a few underlying sockets; and they will be opened and primed as soon as the browser is there. 'Our' code that uses .Net WebClient/WebRequest is totally inefficient in comparison, as everything is initialised anew each time.

• There are a lot of platform resources associated with networking, and whilst .Net makes it much easier to code with networks, it's still bound to the same platform resource issues. Ergo, the closer you are to the platform you are, the faster some code will be. IE and Firefox et al are native and therefore can thrown around system resources natively; .Net isn't and therefore some marshalling(=slow) is required to set things up. Obviously, once a port is opened and being used, however, .Net is still no slouch; but it almost would never be as fast as well-written non-marshalled native code.