System.Net.WebClient unreasonably slow

Question

When using the System.Net.WebClient.DownloadData() method I\'m getting an unreasonably slow response time.

When fetching an url using the WebClient class in .NET it

Answer 1

Another alternative (also free) to Wireshark is Microsoft Network Monitor.

Answer 2

I had that problem with WebRequest. Try setting Proxy = null;

    WebClient wc = new WebClient();
    wc.Proxy = null;

By default WebClient, WebRequest try to determine what proxy to use from IE settings, sometimes it results in like 5 sec delay before the actual request is sent.

This applies to all classes that use WebRequest, including WCF services with HTTP binding. In general you can use this static code at application startup:

WebRequest.DefaultWebProxy = null;

Answer 3

Another cause for extremely slow WebClient downloads is the destination media to which you are downloading. If it is a slow device like a USB key, this can massively impact download speed. To my HDD I could download at 6MB/s, to my USB key, only 700kb/s, even though I can copy files to this USB at 5MB/s from another drive. wget shows the same behavior. This is also reported here:

https://superuser.com/questions/413750/why-is-downloading-over-usb-so-slow

So if this is your scenario, an alternative solution is to download to HDD first and then copy files to the slow medium after download completes.

Answer 4

Download Wireshark here http://www.wireshark.org/

Capture the network packets and filter the "http" packets. It should give you the answer right away.

Answer 5

WebClient may be slow on some workstations when Automatic Proxy Settings in checked in the IE settings (Connections tab - LAN Settings).

Answer 6

There is nothing inherently slow about .NET web requests; that code should be fine. I regularly use WebClient and it works very quickly.

How big is the payload in each direction? Silly question maybe, but is it simply bandwidth limitations?

IMO the most likely thing is that your web-site has spun down, and when you hit the URL the web-site is slow to respond. This is then not the fault of the client. It is also possible that DNS is slow for some reason (in which case you could hard-code the IP into your "hosts" file), or that some proxy server in the middle is slow.

If the web-site isn't yours, it is also possible that they are detecting atypical usage and deliberately injecting a delay to annoy scrapers.

I would grab Fiddler (a free, simple web inspector) and look at the timings.