C# generate a cookie dynamically that site will accept?

Question

I have tried just about all related solutions found on the Web, but they all refused to work for some reason. And this does not work too: C# - HttpWebRequest POST (Login to Face

Answer 1

It is a little bit more complicated than at first sight expected. The browser makes actually two calls. The first one returns an html script with a small piece of javascript that when executed sets a cookie and reload the page. In your c# code you have to mimic that.

In your form class add an instance variable to hold all the cookies across multiple httpwebrequest calls:

readonly CookieContainer cookiecontainer = new CookieContainer();

I have created a Builder method that creates the HttpWebRequest and returns an HttpWebResponse. It takes a namevaluecollection to add any cookies to the Cookiecontainer.

private HttpWebResponse Builder(string url, string host, NameValueCollection cookies)
{
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
    request.Method = WebRequestMethods.Http.Get;
    request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
    // _request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip,deflate,sdch");
    request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.8");
    request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");

    request.Host = host;
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36";

    request.CookieContainer = cookiecontainer;

    if (cookies != null)
    {
        foreach (var cookiekey in cookies.AllKeys)
        {
            request.CookieContainer.Add(
                new Cookie(
                    cookiekey, 
                    cookies[cookiekey],
                    @"/",
                    host));    
        }
    }
    return (HttpWebResponse) request.GetResponse();
}

If the incoming stream turns out to be an text/html contenttype we need to parse its content and return the cookie name and value. The Parse method does just that:

// find in the html and return the three parameters in a string array
// setCookie('YPF8827340282Jdskjhfiw_928937459182JAX666', '127.0.0.1', 10);
private static string[] Parse(Stream _stream, string encoding)
{
    const string setCookieCall = "setCookie('";

    // copy html as string
    var ms = new MemoryStream();
    _stream.CopyTo(ms);
    var html = Encoding.GetEncoding(encoding).GetString(ms.ToArray());

    // find setCookie call
    var findFirst = html.IndexOf(
        setCookieCall, 
        StringComparison.InvariantCultureIgnoreCase) + setCookieCall.Length;
    var last = html.IndexOf(");", findFirst, StringComparison.InvariantCulture);

    var setCookieStatmentCall = html.Substring(findFirst, last - findFirst);
    // take the parameters
    var parameters = setCookieStatmentCall.Split(new[] {','});
    for (int x = 0; x < parameters.Length; x++)
    {
        // cleanup
        parameters[x] = parameters[x].Replace("'", "").Trim();
    }
    return parameters;
}

Now are our building blocks complete we can start calling our methods from the Click method. We use a loop to call our Builder twice to obtain a result from the given url. Based on the received contenttype we either Parse or create the Image from the stream.

private void button1_Click(object sender, EventArgs e)
{
    var cookies = new NameValueCollection();
    for (int tries = 0; tries < 2; tries++)
    {
        using (var response = Builder(_URL, "www.habbo" + _Country, cookies))
        {
            using (var stream = response.GetResponseStream())
            {
                string contentType = response.ContentType.ToLowerInvariant();
                if (contentType.StartsWith("text/html"))
                {
                    var parameters = Parse(stream, response.CharacterSet);
                    cookies.Add(parameters[0], parameters[1]);
                }
                if (contentType.StartsWith("image"))
                {
                    pictureBox1.Image = Image.FromStream(stream);
                    break; // we're done, get out
                }
            }
        }
    }
}

Words of caution

This code works for the url in your question. I didn't take any measures to handle other patterns, and/or exceptions. It is up to you to add that. Also when doing this kind of scraping make sure the owner of the website does allow this.