Multithreaded WebClient requests return error - System.Net.WebException

Question

I have 5000+ pages I want to download using WebClient. Since I want that done as fast as possible I am trying to use multithreading (using BlockingCollection<

Answer 1

The class is not guaranteed to be thread safe. from MSDN:

Any instance members are not guaranteed to be thread safe

Update

Use one HttpWebRequest for each request that you make. If you make a lot of requests to different web sites it doesn't matter if you use WebClient or HttpWebRequest.

If you do a lot of requests to the same web site it is still not as inefficient as it seems. HttpWebRequest reuse connections (it's hidden underneath the hood). Microsoft uses something called service points and you can access them through the HttpWebRequest.ServicePoint property. If you click on the property definition you come to the ServicePoint documentation where you can fine tune the number of connections per web site etc.