Android HttpUrlConnection EOFException

Question

I would like to know if there are known issues on Android with HttpUrlConnection and POST requests. We are experiencing intermittent EOFExceptions when maki

Answer 1

HttpURLConnection library internally maintains a pool of Connections. So, whenever a request is send, it first checks if there is an existing connection already present in the pool, based on which it decides to create a new one.

These connections are nothing but sockets, and this library by default does not closes these sockets. It may sometimes happen that a connection (socket) which is not currently being used and is present in the pool is no longer usable as the Server may choose to terminate the connection after some time. Now, since the connection even though is closed by the server, the library does not knows about it and assumes the connection/socket to be still connected. Thus it sends the new request using this stale connection and hence we get EOFException.

The best way to handle this is to check the Response Headers after each request you send. The server always sends a "Connection: Close" before terminating a connection (HTTP 1.1). So, you can use getHeaderField() and check for "Connection" field. Another thing to note is that server ONLY sends this connection field when it is about to terminate the connection. So, you need to code around this with the possibility of getting a "null" in the normal case (when server is not closing the connection)

Answer 2

This worked for me.

public ResponseObject sendPOST(String urlPrefix, JSONObject payload) throws JSONException {
    String line;
    StringBuffer jsonString = new StringBuffer();
    ResponseObject response = new ResponseObject();
    try {

        URL url = new URL(POST_URL);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setDoInput(true);
        connection.setDoOutput(true);
        connection.setReadTimeout(10000);
        connection.setConnectTimeout(15000);
        connection.setRequestMethod("POST");
        connection.setRequestProperty("Accept", "application/json");
        connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");

        OutputStream os = connection.getOutputStream();
        os.write(payload.toString().getBytes("UTF-8"));
        os.close();
        BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        while ((line = br.readLine()) != null) {
            jsonString.append(line);
        }
        response.setResponseMessage(connection.getResponseMessage());
        response.setResponseReturnCode(connection.getResponseCode());
        br.close();
        connection.disconnect();
    } catch (Exception e) {
        Log.w("Exception ",e);
        return response;
    }
    String json = jsonString.toString();
    response.setResponseJsonString(json);
    return response;
}

Answer 3

connection.addRequestProperty("Accept-Encoding", "gzip");

is the answer

Answer 4

Our conclusion is that there is an issue in the Android platform. Our workaround was to catch the EOFException and retry the request N number of times. Below is the pseudo code:

private static final int MAX_RETRIES = 3;

private ResponseType fetchResult(RequestType request) {
    return fetchResult(request, 0);
}

private ResponseType fetchResult(RequestType request, int reentryCount) {
    try {
        // attempt to execute request
    } catch (EOFException e) {
        if (reentryCount < MAX_RETRIES) {
            fetchResult(request, reentryCount + 1);
        }
    }
    // continue processing response
}

Answer 5

This workaround tends to be reliable and performant:

static final int MAX_CONNECTIONS = 5;

T send(..., int failures) throws IOException {
    HttpURLConnection connection = null;

    try {
        // initialize connection...

        if (failures > 0 && failures <= MAX_CONNECTIONS) {
            connection.setRequestProperty("Connection", "close");
        }

        // return response (T) from connection...
    } catch (EOFException e) {
        if (failures <= MAX_CONNECTIONS) {
            disconnect(connection);
            connection = null;

            return send(..., failures + 1);
        }

        throw e;
    } finally {
        disconnect(connection);
    }
}

void disconnect(HttpURLConnection connection) {
    if (connection != null) {
        connection.disconnect();
    }
}

This implementation relies on the fact that the default number of connections that can be opened with a server is 5 (Froyo - KitKat). This means that up to 5 stale connections may exist, each of which will have to be closed.

After each failed attempt, the Connection:close request property will cause the underlying HTTP engine to close the socket when connection.disconnect() is called. By retrying up to 6 times (max connections + 1), we ensure that the last attempt will always be given a new socket.

The request may experience additional latency if no connections are alive, but that is certainly better than an EOFException. In that case, the final send attempt won't immediately close the freshly opened connection. That's the only practical optimization that can be made.

Instead of relying on the magic default value of 5, you may be able to configure the system property yourself. Keep in mind that this property is accessed by a static initializer block in KitKat's ConnectionPool.java, and it works like this in older Android versions too. As a result, the property may be used before you have a chance to set it.

static final int MAX_CONNECTIONS = 5;

static {
    System.setProperty("http.maxConnections", String.valueOf(MAX_CONNECTIONS));
}

Answer 6

I suspect it might be the server that is at fault here, and the HttpURLConnection is not as forgiving as other implementations. That was the cause of my EOFException. I suspect in my case this would not be intermittent (fixed it before testing the N retry workaround), so the answers above relate to other issues and be a correct solution in those cases.

My server was using python SimpleHTTPServer and I was wrongly assuming all I needed to do to indicate success was the following:

self.send_response(200)

That sends the initial response header line, a server and a date header, but leaves the stream in the state where you are able to send additional headers too. HTTP requires an additional new line after headers to indicate they are finished. It appears if this new line isn't present when you attempt to get the result body InputStream or response code etc with HttpURLConnection then it throws the EOFException (which is actually reasonable, thinking about it). Some HTTP clients did accept the short response and reported the success result code which lead to me perhaps unfairly pointing the finger at HttpURLConnection.

I changed my server to do this instead:

self.send_response(200)
self.send_header("Content-Length", "0")
self.end_headers()

No more EOFException with that code.