Remote server (403) Forbidden error while using WebClient

Question

I am trying to query some URL using WebClient.

I have a collection which I loop through to get the QueryString value, and build final URL, and then pass it on to client.

The very first time it gets executed well, I get proper response, however, when it goes in loop second time, I get the error:

System.Net.WebException --> The remote server returned an error: (403) Forbidden.

If I get response for very first time. Then I should get for rest of the collection too.

Any clue why? What I might be missing?

Below is the code snippet I am using.

 using(System.IO.StreamWriter file = new System.IO.StreamWriter(@"C:\sample.text")) {
    foreach(var f in fileCollections) {
        strFinalURL = string.Empty;

        strFinalURL = "someURL" + f; // f can be considered as querystring param value

        try {
            using(var client = new WebClient()) {

                test = client.DownloadString(strFinalURL);
                if (!test.Contains("somecondition")) {
                    file.WriteLine("");
                }
            }
        } catch (System.Exception ex) {
            Console.WriteLine(ex.Message);
        }
    }
}

Answer 1

If everything you said is correct, then I suspect server throttling is playing a part. Try adding a Thread.Sleep in your loop to slow things down.

Answer 2

Some web servers can block requests based on the user agent string they provide. In your case, you send an empty string as a user agent. Try to add the user agent of a browser, any browser would be fine.

For example:

client.Headers.Add("user-agent", " Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");

For a list of user agents, you can click here.