How to log request/response using java.net.http.HttpClient?

Question

The HttpClient introduced experimentally in Java 9 is now stable in Java 11, but not surprisingly, very few projects seem to actually use it. Documentation is almost non-existing.

One of the most commons asks while making a HTTP call is logging of request/response. How would you do that using the HttpClient, without of course, logging it manually in every single call? Is there an interceptor mechanism like that offered by all other HTTP clients?

Answer 1

If we look at jdk.internal.net.http.common.DebugLogger source code we can see a few loggers using System.Logger, which in turn will useSystem.LoggerFinder to select the logger framework. JUL is the default choice. The logger names are:

• jdk.internal.httpclient.debug
• jdk.internal.httpclient.websocket.debug
• jdk.internal.httpclient.hpack.debug

They can be enabled by setting them as a system property. For example running with -Djdk.internal.httpclient.debug=true will produce:

DEBUG: [main] [147ms] HttpClientImpl(1) proxySelector is sun.net.spi.DefaultProxySelector@6dde5c8c (user-supplied=false)
DEBUG: [main] [183ms] HttpClientImpl(1) ClientImpl (async) send https://http2.github.io/ GET
DEBUG: [main] [189ms] Exchange establishing exchange for https://http2.github.io/ GET,
     proxy=null
DEBUG: [main] [227ms] PlainHttpConnection(?) Initial receive buffer size is: 43690
DEBUG: [main] [237ms] PlainHttpConnection(SocketTube(1)) registering connect event
DEBUG: [HttpClient-1-SelectorManager] [239ms] SelectorAttachment Registering jdk.internal.net.http.PlainHttpConnection$ConnectEvent@354bf356 for 8 (true)
...

Answer 2

You can log request and responses by specifying -Djdk.httpclient.HttpClient.log=requests on the Java command line.

As for testing/mocking you might want to have a look at the offline test: http://hg.openjdk.java.net/jdk/jdk/file/tip/test/jdk/java/net/httpclient/offline/

Depending on what you are looking to achieve you could use a "DelegatingHttpClient" to intercept and log requests and responses too.

Besides the Java API documentation there's also some high level documentation at http://openjdk.java.net/groups/net/httpclient/index.html

Additional note:

The jdk.httpclient.HttpClient.log property is an implementation specific property whose value is a comma separated list which can be configured on the Java command line for diagnosis/debugging purposes with the following values:

-Djdk.httpclient.HttpClient.log=
       errors,requests,headers,
       frames[:control:data:window:all],content,ssl,trace,channel

Answer 3

On our side, we did not find the logging provided by -Djdk.internal.httpclient.debug readable enough. The solution we came up with is to wrap the HttpClient with a decorator that will be able to intercept the calls and provide logging. Here how it somehow looks (should be done not only for send but sendAsync methods) :

public class HttpClientLoggingDecorator extends HttpClient {

  private static final Logger logger = Logger.getLogger(HttpClientLoggingDecorator.class.getName());

  private final HttpClient client;

  ...

  @Override
  public <T> HttpResponse<T> send(HttpRequest req, HttpResponse.BodyHandler<T> responseBodyHandler)
    throws IOException,
      InterruptedException
  {
    subscribeLoggerToRequest(req);

    HttpResponse<T> response = client.send(req, responseBodyHandler);

    logResponse(response);
    return response;
  }

  private void subscribeLoggerToRequest(HttpRequest req) {
    // define a consumer for how you want to log
    // Consumer<String> bodyConsumer = ...;
    if (req.bodyPublisher().isPresent()) {
      req.bodyPublisher()
              .ifPresent(bodyPublisher -> bodyPublisher.subscribe(new HttpBodySubscriber(bodyConsumer)));
    } else {
      bodyConsumer.accept(NO_REQUEST_BODY);
    }
  }

  private <T> void logResponse(HttpResponse<T> response) {
    // String responseLog = ...;
    logger.info(responseLog);
  }

}

And here is the HttpBodySubscriber:

public class HttpBodySubscriber implements Flow.Subscriber<ByteBuffer> {

  private static final long UNBOUNDED = Long.MAX_VALUE;

  private final Consumer<String> logger;

  public HttpBodySubscriber(Consumer<String> logger) {
    this.logger = logger;
  }

  @Override
  public void onSubscribe(Flow.Subscription subscription) {
    subscription.request(UNBOUNDED);
  }

  @Override
  public void onNext(ByteBuffer item) {
    logger.accept(new String(item.array(), StandardCharsets.UTF_8));
  }

  @Override
  public void onError(Throwable throwable) {
  }

  @Override
  public void onComplete() {
  }

}