How to log request/response using java.net.http.HttpClient?

Question

The HttpClient introduced experimentally in Java 9 is now stable in Java 11, but not surprisingly, very few projects seem to actually use it. Documentation is almost non-existing.

One of the most commons asks while making a HTTP call is logging of request/response. How would you do that using the HttpClient , without of course, logging it manually in every single call? Is there an interceptor mechanism like that offered by all other HTTP clients?

Answer 1

You can log request and responses by specifying -Djdk.httpclient.HttpClient.log=requests on the Java command line.

As for testing/mocking you might want to have a look at the offline test: http://hg.openjdk.java.net/jdk/jdk/file/tip/test/jdk/java/net/httpclient/offline/

Depending on what you are looking to achieve you could use a "DelegatingHttpClient" to intercept and log requests and responses too.

Besides the Java API documentation there's also some high level documentation at http://openjdk.java.net/groups/net/httpclient/index.html

Additional note:

The jdk.httpclient.HttpClient.log property is an implementation specific property whose value is a comma separated list which can be configured on the Java command line for diagnosis/debugging purposes with the following values:

-Djdk.httpclient.HttpClient.log=
       errors,requests,headers,
       frames[:control:data:window:all],content,ssl,trace,channel,all

Answer 2

If we look at jdk.internal.net.http.common.DebugLogger source code we can see a few loggers using System.Logger , which in turn will use System.LoggerFinder to select the logger framework. JUL is the default choice. The logger names are:

• jdk.internal.httpclient.debug
• jdk.internal.httpclient.websocket.debug
• jdk.internal.httpclient.hpack.debug

They can be enabled by setting them as a system property. For example running with -Djdk.internal.httpclient.debug=true will produce:

DEBUG: [main] [147ms] HttpClientImpl(1) proxySelector is sun.net.spi.DefaultProxySelector@6dde5c8c (user-supplied=false)
DEBUG: [main] [183ms] HttpClientImpl(1) ClientImpl (async) send https://http2.github.io/ GET
DEBUG: [main] [189ms] Exchange establishing exchange for https://http2.github.io/ GET,
     proxy=null
DEBUG: [main] [227ms] PlainHttpConnection(?) Initial receive buffer size is: 43690
DEBUG: [main] [237ms] PlainHttpConnection(SocketTube(1)) registering connect event
DEBUG: [HttpClient-1-SelectorManager] [239ms] SelectorAttachment Registering jdk.internal.net.http.PlainHttpConnection$ConnectEvent@354bf356 for 8 (true)
...

Answer 3

On our side, we did not find the logging provided by -Djdk.internal.httpclient.debug readable enough. The solution we came up with is to wrap the HttpClient with a decorator that will be able to intercept the calls and provide logging. Here how it somehow looks (should be done not only for send but sendAsync methods) :

public class HttpClientLoggingDecorator extends HttpClient {

  private static final Logger logger = Logger.getLogger(HttpClientLoggingDecorator.class.getName());

  private final HttpClient client;

  ...

  @Override
  public <T> HttpResponse<T> send(HttpRequest req, HttpResponse.BodyHandler<T> responseBodyHandler)
    throws IOException,
      InterruptedException
  {
    subscribeLoggerToRequest(req);

    HttpResponse<T> response = client.send(req, responseBodyHandler);

    logResponse(response);
    return response;
  }

  private void subscribeLoggerToRequest(HttpRequest req) {
    // define a consumer for how you want to log
    // Consumer<String> bodyConsumer = ...;
    if (req.bodyPublisher().isPresent()) {
      req.bodyPublisher()
              .ifPresent(bodyPublisher -> bodyPublisher.subscribe(new HttpBodySubscriber(bodyConsumer)));
    } else {
      bodyConsumer.accept(NO_REQUEST_BODY);
    }
  }

  private <T> void logResponse(HttpResponse<T> response) {
    // String responseLog = ...;
    logger.info(responseLog);
  }

}

And here is the HttpBodySubscriber :

public class HttpBodySubscriber implements Flow.Subscriber<ByteBuffer> {

  private static final long UNBOUNDED = Long.MAX_VALUE;

  private final Consumer<String> logger;

  public HttpBodySubscriber(Consumer<String> logger) {
    this.logger = logger;
  }

  @Override
  public void onSubscribe(Flow.Subscription subscription) {
    subscription.request(UNBOUNDED);
  }

  @Override
  public void onNext(ByteBuffer item) {
    logger.accept(new String(item.array(), StandardCharsets.UTF_8));
  }

  @Override
  public void onError(Throwable throwable) {
  }

  @Override
  public void onComplete() {
  }

}