.NET HttpWebRequest Speed versus Browser

Question

I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).

If I use HttpWebRequest to request an html page

Answer 1

It maybe that bbc.co.uk checks the User-Agent header that is being passed to it and handles the response based on that. So if it sees automated clients then it responds slowly, where as if it believes that there is a real person at the end of the line then it speeds up. If you really want to try it out just tell the HttpWebRequest to pass a different header.