java httpclient connectionpool租期与保持活动状态

Question

I have used apache httpclient 4.5 in production for a while now, but recently, with the addition of a new use case, the system started failing.

We have multiple services that communicate through REST webservices, the client is a wrapper around apache httpclient 4.5.

Say i have service A communicating with service B. The communication works correctly until I restart service B. The next call I initiate from service A to service B fails, due to time out. After doing some research I found that the underlying TCP connection is reused for performance reasons (no more 2 way handshake etc). Since the server has been restarted, the underlying TCP connection is stale.

After reading the documentation, I found out that I can expire my connection after n seconds. Say I restart service B, then the call will fail the first n seconds, but after that the connection is rebuild. This is the keepAliveStrategy I implemented

    connManager = new PoolingHttpClientConnectionManager();
    connManager.setMaxTotal(100);
    connManager.setDefaultMaxPerRoute(10);
    ConnectionKeepAliveStrategy keepAliveStrategy = new DefaultConnectionKeepAliveStrategy() {

        public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
            long keepAliveDuration = super.getKeepAliveDuration(response, context);
            if (keepAliveDuration == -1) {
                keepAliveDuration = 45 * 1000; // 45 seconds
            }
            return keepAliveDuration;
        }
    };
    CloseableHttpClient closeableHttpClient = HttpClients.custom()
        .setConnectionManager(connManager)
        .setKeepAliveStrategy(keepAliveStrategy)
        .build();

I am just wondering if this is correct usage of this library. I this the way it is meant to work or am I making everything overly complex?

Answer 1

Not sure it's 100% the same scenario, but here's my 2 cents:

We had a similar issues (broken connections in pool after a period of inactivity). When we were using an older version of HttpClient (3.X), we used the http.connection.stalecheck manager parameter, taking a minor performance hit over the possibility to get a IOException when a connection has been used that was closed server-side.

After upgrading to 4.4+ this approach was deprecated and started using setValidateAfterInactivity , which is a middle ground between per-call validation and runtime-error scenario:

PoolingHttpClientConnectionManager poolingConnManager = new PoolingHttpClientConnectionManager();
poolingConnManager.setValidateAfterInactivity(5000);

void oahicPoolingHttpClientConnectionManager.setValidateAfterInactivity(int ms)

Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to being leased to the consumer. Non-positive value passed to this method disables connection validation. This check helps detect connections that have become stale (half-closed) while kept inactive in the pool.

If you're also controlling the consumed API, you can adapt the keep-alive strategy to the timing your client uses. We're using AWS Cloudfront + ELB's with connection draining for deregistered instances to ensure the kept-alive connections are fully closed, when performing a rolling upgrade. I guess as long as the connections are guaranteed to be kept alive for, say 30 seconds, any value passed to the connection manager below that will always ensure the validity check will mitigate any runtime I/O errors which are purely related to stale/expired connections.