.NET HttpWebRequest Speed versus Browser

Question

I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).

If I use HttpWebRequest to request an html page

Answer 1

Whenever you measure anything, you have to account for the startup costs. If your .net code is in a single process,and you are only measuring the single request, then your measurement will be tainted by first time costs of initializing assemblies, types, etc.

As Darin and others have suggested, you should make sure that:

1) You are not running the process under debuggger. 2) You account for startup costs.

One way you can do #2, is to make two requests and only measure the second one. Or you can make N requests, discard the 1st one, and get the average of last N-1 requests. Also make sure that you read the entity stream.

Answer 2

Have you watched the network while using the browser? Perhaps the browser is using cached resources?

Answer 3

It maybe that bbc.co.uk checks the User-Agent header that is being passed to it and handles the response based on that. So if it sees automated clients then it responds slowly, where as if it believes that there is a real person at the end of the line then it speeds up. If you really want to try it out just tell the HttpWebRequest to pass a different header.

Answer 4

What's the breakdown of that 1.7s? I suspect you are measuring the entire process?

Using this piece of code I get about 200ms in average:

var request = (HttpWebRequest)WebRequest.Create("http://www.bbc.co.uk/news/");

var stopwatch = new Stopwatch();
stopwatch.Start();

using (var response = (HttpWebResponse)request.GetResponse())
{
    stopwatch.Stop();
    Console.WriteLine("Elapsed: {0}ms", stopwatch.ElapsedMilliseconds);

    var responseStream = response.GetResponseStream();
    if (responseStream != null)
        using (var sr = new StreamReader(responseStream))
            Console.WriteLine("Title: {0}", Regex.Match(sr.ReadToEnd(), @"title>(.*)</title").Groups[1].Value);
}

Edit changed the code just to measure the actual HTTP request and tried again using Fiddler as well:

Program above: Elapsed: 78ms

Fiddler: Overall Elapsed: 00:00:00.0620000

Answer 5

Markos' answer worked perfectly for me for the same issue:

request.Proxy = new WebProxy();

Reduced a 16-second request to less than a second. Thanks!

Answer 6

I'd jack Fiddler in the middle, run the browser request and the .NET request one after the other and make sure you're really getting what you think. It's possible there's redirection or something else hinky going on (maybe browser is pre-appending the '/' while .NET waits for the redir, etc) that isn't immediately visible. I've built huge apps on the .NET HTTP client with nothing like what you describe- something else must be going on.

What happens if you stick '/' on the end of the URL?