Why Does my HttpWebRequest Return 400 Bad request?

Question

The following code fails with a 400 bad request exception. My network connection is good and I can go to the site but I cannot get this uri with HttpWebRequest.

Answer 1

Set UserAgent and Referer in your HttpWebRequest:

var request = (HttpWebRequest)WebRequest.Create(@"http://www.youtube.com/");
request.Referer = "http://www.youtube.com/"; // optional
request.UserAgent =
    "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; " +
    "Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; " +
    ".NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618; " +
    "InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)";
try
{
    var response = (HttpWebResponse)request.GetResponse();
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
        var html = reader.ReadToEnd();
    }
}
catch (WebException ex)
{
    Log(ex);
}

Answer 2

First, cast the WebRequest to an HttpWebRequest like this:

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(@"http://www.youtube.com/");

Then, add this line of code:

req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

Answer 3

Maybe you've got a proxy server running, and you haven't set the Proxy property of the HttpWebRequest?

Answer 4

There could be many causes for this problem. Do you have any more details about the WebException?

One cause, which I've run into before, is that you have a bad user agent string. Some websites (google for instance) check that requests are coming from known user agents to prevent automated bots from hitting their pages.

In fact, you may want to check that the user agreement for YouTube does not preclude you from doing what you're doing. If it does, then what you're doing may be better accomplished by going through approved channels such as web services.