Connections leaking with state CLOSE_WAIT with HttpClient

问题

We are using JDK11 java.net.http HTTP client to get data from an API. After we receive the response the connections remain opened in our server with TCP state CLOSE_WAIT, which means the client must close the connection.

From RFC 793 terminology:

CLOSE-WAIT - represents waiting for a connection termination request from the local user.

This is our client code which runs on WildFly 16 running on Java 12 as a stateless REST API. We don't understand why this is happening.

import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpClient.Version;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;

public class SocketSandbox {

    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newBuilder().version(Version.HTTP_1_1).build();
        try (var listener = new ServerSocket(59090)) {
            System.out.println("Server is running...");
            while (true) {
                try (var socket = listener.accept()) {
                    HttpRequest request = HttpRequest
                            .newBuilder(URI.create("<remote_URL>"))
                            .header("content-type", "application/json").build();
                    HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
                    var out = new PrintWriter(socket.getOutputStream(), true);
                    out.println(String.format("Response HTTP status: %s", response.statusCode()));
                }
            }
        }
    }

}

We get the "status code" meaning the http response was processed.

When using the same code to call other endpoints connections are fine. It seems to be a particular issue with the remote API we are calling, but still we don't understand why the Java HTTP client is keeping connections opened.

We tried both Windows and Linux machines, and even standalone outside of WildfFly, but the same result happens. After each request, even doing it from our stateless client and receiving the response, each one is left as CLOSE_WAIT and never close.

Connections will disappear if we shutdown the Java process.

Headers that are sent by the HTTP client:

connection: 'Upgrade, HTTP2-Settings','content-length': '0',
host: 'localhost:3000', 'http2-settings': 'AAEAAEAAAAIAAAABAAMAAABkAAQBAAAAAAUAAEAA',
upgrade: 'h2c',
user-agent': 'Java-http-client/12'

Server returns response with header: Connection: close

Update (1)

We tried to fine-tune the pool parameters in implementation class jdk.internal.net.http.ConnectionPool.

It did not solve the problem.

System.setProperty("jdk.httpclient.keepalive.timeout", "5"); // seconds
System.setProperty("jdk.httpclient.connectionPoolSize", "1");

Update (2)

With Apache HTTP the connections l getting left in CLOSE_WAIT state for about 90 seconds, but it is able to the connections after that time.

Calling method HttpGet.releaseConnection() force the connection close immediately.

HttpClient client = HttpClients.createDefault();
HttpGet get = new HttpGet("https://<placeholderdomain>/api/incidents/list");
get.addHeader("content-type", "application/json");
HttpResponse response = client.execute(get);

// This right here did the trick
get.releaseConnection();

return response.getStatusLine().getStatusCode();

And with OkHttp client it worked as expected out of the box, no connections stuck.

OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
        .url("https://<placeholderdomain>/grb/sb/incidentes/listar")
        .header("content-type", "application/json").build();
Response response = client.newCall(request).execute();
return response.body().string();

We are still trying to find how to make it work in java-http-client so that we don't have to rewrite the code.

回答1:

I wouldn't recommend creating a new client for every new request. This is defeating the purpose of HTTP/2 which allows multiplexing requests on a single connection.

The second thing is that the two properties:

System.setProperty("jdk.httpclient.keepalive.timeout", "5"); // seconds
System.setProperty("jdk.httpclient.connectionPoolSize", "1");

only apply to HTTP/1.1 connections, not HTTP/2. Also take care that these properties are only read once at class loading time. So setting them after the java.net.http classes have been loaded will have no effect.

Finally it may take some time after an HttpClient is released before all kept alive connections are closed - the internal mechanism to do so is basically relying on GC - and this is not very friendly with short lived HttpClients.