httpWebRequest (The underlying connection was closed: The connection was closed unexpectedly.)

Question

I am developing an C# application which logs data from a webserver. It sends the following post request to the webserver and awaits for the response.

    /// <summary>
    /// Function for obtaining testCgi data 
    /// </summary>
    /// <param name="Parameters"></param>
    /// <returns></returns>
    private string HttpmyPost(string Parameters)
    {
        string str = "No response";
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriTestCGI);
        request.Method = "POST";

        byte[] bytes = Encoding.UTF8.GetBytes(Parameters);
        request.ContentLength = bytes.Length;

        Stream requestStream = request.GetRequestStream();
        requestStream.Write(bytes, 0, bytes.Length);
        requestStream.Close();

            WebResponse response = request.GetResponse();
            Stream stream = response.GetResponseStream();
            StreamReader reader = new StreamReader(stream);

            try
            {
                var result = reader.ReadToEnd();
            stream.Dispose();
            str = result.ToString();
            reader.Dispose();
        }
        catch (WebException ex)
        {
            //System.Windows.Forms.MessageBox.Show(ex.Message);
            System.Diagnostics.Trace.WriteLine(ex.Message);

        }
        finally
        {
            request.Abort();
        }
        return str;
    }

I am getting the error

> "The underlying connection was closed: The connection was closed
> unexpectedly"

I have tried to debug the error, and I have used fiddler in order to check the post request as given from Firefox. To my surprise whenever Fiddler was my program was working perfectly. When I close fiddler I am having the same error.

I suspect that since Fiddler is acting as a proxy it may change some of the settings. I have tried using webclient and the result was the same.

When I tried coding the request in python, everything worked as it should without no problem. Of cource I have the option of installing IronPython and wrapping that particular function, however I consider this overkill and lacking elegance, so I am pursuing a leaner approach. I suspect this is nothing more that a setting adjustment.

I have tried modifying and in my case it is indifferent.

request.Accept 
request.ReadWriteTimeout 
request.Timeout 
request.UserAgent 
request.Headers
request.AutomaticDecompression 
request.Referer
request.AllowAutoRedirect
//request.TransferEncoding 
request.Expect
request.ServicePoint.Expect100Continue 
request.PreAuthenticate 
request.KeepAlive 
request.ProtocolVersion 
request.ContentType

With or without the above adjustments the code works when Fiddler is capturing data.

Also it might be noteworthy that the program yields the error at

WebResponse response = request.GetResponse();

UPDATE: Following @EricLaw suggestions I looked into Latency. I found this article HttpWebRequest gets slower when adding an Interval which suggested turning of the Nagle Algorithm. Now there are no closed connections, although there is a small lag in the overall response (when I use winforms, and not async).

Answer 1

I write a bit about how Fiddler can "magically" fix things here: http://blogs.telerik.com/fiddler/posts/13-02-28/help!-running-fiddler-fixes-my-app-

The issue you're encountering is actually a bug in the .NET Framework itself. The rules of HTTP are such that the server may close a KeepAlive connection at any time after sending the first response (e.g. it doesn't need to accept another request on the connection, even if the client requested KeepAlive behavior).

.NET has a bug where it expects that the server will include a Connection: close response header if it will close the connection after the response is complete. If the server closes the connection without the Connection: Close header (entirely valid per RFC2616), .NET will encounter the closed connection when attempting to send the next request on the connection and it will throw this exception. What .NET should be doing is silently creating a new connection and resending the request on that new connection.

Fiddler resolves this problem because it doesn't care if the server closes the connection, and it keeps the connection to the client alive. When the client sends its second request, Fiddler attempts to reuse its connection to the server, notices that it's closed, and silently creates a new connection.

You can mitigate this problem in your code by:

1. Disabling keepalive on the request (this hurts performance)
2. Catching the exception and retrying automatically
3. Changing the server to keep connections alive longer

Approach #3 only works if you control the server and because the client may be behind a gateway/proxy that closes connections after use, you should probably use approach #2 as well.

Answer 2

a suggestion and a question: 1) if you really want to see what's going on install Wireshark. It will show you exactly what's being sent / received. and it will make it possible to compare with the fiddler. I guess you are missing a header, like request.ContentType = "...." but only wireshark will show you which one (is sent via your working alternative, while not sent by your HttpWebRequest).

2) are you getting the error inside the http response content, or is it an exception, and if it's an exception is it caught in your catch, or does it occur during the request, before your try statement?

Answer 3

Fiddler works as an Internet Proxy. If your code works while Fiddler is running, (and maybe also from a browser), then you may have a problem with your proxy settings.