Java8 Apache Http Client 4.5.12 - SocketException

Question

Below is my ApacheHttpClient which is a Spring bean

@Service
public class ApacheHttpClient implements IHttpClient {
    private static final Logger              LOGGER                                             = Logger
            .getInstance(ApacheHttpClient.class);
    private static final int                 DEFAULT_MAX_TOTAL_CONNECTIONS                      = 400;
    private static final int                 DEFAULT_IDLE_CONNECTION_EVICTION_FREQUENCY_SECONDS = 300;
    private static final int                 DEFAULT_MAX_CONNECTIONS_PER_ROUTE                  = DEFAULT_MAX_TOTAL_CONNECTIONS;

    private static final int                 DEFAULT_CONNECTION_TIMEOUT_MILLISECONDS            = (60 * 1000);
    private static final int                 DEFAULT_READ_TIMEOUT_MILLISECONDS                  = (4 * 60 * 1000);
    private static final int                 DEFAULT_WAIT_TIMEOUT_MILLISECONDS                  = (60 * 1000);
    private static final int                 DEFAULT_VALIDATE_AFTER_INACTIVITY_MILLISECONDS     = (5 * 60 * 1000);

    private static final int                 DEFAULT_KEEP_ALIVE_MILLISECONDS                    = (5 * 60 * 1000);
    private static final int                 DEFAULT_REQUEST_RETRY                              = 2;

    @Autowired
    private CPSSLContextHelper               cpSSLContext;

    @Autowired
    private CollaborationPortalConfiguration cpConfiguration;

    private int                              keepAlive                                          = DEFAULT_KEEP_ALIVE_MILLISECONDS;
    private int                              maxTotalConnections                                = DEFAULT_MAX_TOTAL_CONNECTIONS;

    private int                              maxConnectionsPerRoute                             = DEFAULT_MAX_CONNECTIONS_PER_ROUTE;
    private int                              connectTimeout                                     = DEFAULT_CONNECTION_TIMEOUT_MILLISECONDS;
    private int                              readTimeout                                        = DEFAULT_READ_TIMEOUT_MILLISECONDS;

    private int                              waitTimeout                                        = DEFAULT_WAIT_TIMEOUT_MILLISECONDS;
    private int                              requestRetry                                       = DEFAULT_REQUEST_RETRY;

    private CloseableHttpClient              httpClient;

    private ConnectionKeepAliveStrategy      keepAliveStrategy                                  = (response,
            context) -> {
                                                                                                    HeaderElementIterator it = new BasicHeaderElementIterator(
                                                                                                            response.headerIterator(
                                                                                                                    HTTP.CONN_KEEP_ALIVE));
                                                                                                    while (it
                                                                                                            .hasNext()) {
                                                                                                        HeaderElement he = it
                                                                                                                .nextElement();
                                                                                                        String param = he
                                                                                                                .getName();
                                                                                                        String value = he
                                                                                                                .getValue();
                                                                                                        if (value != null
                                                                                                                && param.equalsIgnoreCase(
                                                                                                                        "timeout")) {
                                                                                                            try {
                                                                                                                return Long
                                                                                                                        .parseLong(
                                                                                                                                value)
                                                                                                                        * 1000;
                                                                                                            } catch (NumberFormatException ignore) {}
                                                                                                        }
                                                                                                    }
                                                                                                    return keepAlive;
                                                                                                };


    @PostConstruct
    public void initializeApacheHttpClient() {

        // config timeout
        RequestConfig config = RequestConfig.custom()
                .setConnectTimeout(connectTimeout)
                .setConnectionRequestTimeout(waitTimeout)
                .setSocketTimeout(readTimeout).build();
        Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory> create()
                .register("http", PlainConnectionSocketFactory.INSTANCE)
                .register("https", new SSLConnectionSocketFactory(customSSLContext.getSSLContext())).build();

        PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);

        connManager.setMaxTotal(maxTotalConnections);

        // Increase default max connection per route
        connManager.setDefaultMaxPerRoute(maxConnectionsPerRoute);

        // Defines period of inactivity in milliseconds after which persistent connections must be re-validated prior to
        // being reused
        connManager.setValidateAfterInactivity(DEFAULT_VALIDATE_AFTER_INACTIVITY_MILLISECONDS);

        httpClient = HttpClients.custom().setKeepAliveStrategy(keepAliveStrategy).setConnectionManager(connManager)
                .setConnectionManagerShared(true).setSSLContext(customSSLContext.getSSLContext())
                .setDefaultRequestConfig(config)
                .setRetryHandler(new DefaultHttpRequestRetryHandler(requestRetry, true))
                .build();

        // detect idle and expired connections and close them
        IdleConnectionEvictor staleMonitor = new IdleConnectionEvictor(connManager, DEFAULT_IDLE_CONNECTION_EVICTION_FREQUENCY_SECONDS);
        staleMonitor.start();

        LOGGER.log(Level.INFO, "Initialize ApacheHttpClient is successful");
    }
}   

Below is my IdleConnectionEvictor

public class IdleConnectionEvictor extends Thread {
    private static final Logger          LOGGER = Logger.getInstance(IdleConnectionEvictor.class);
    ReentrantLock                        lock   = new ReentrantLock();
    private NHttpClientConnectionManager nioConnMgr;

    private int                          httpClientIdleConnectionEvictionFrequency;

    private volatile boolean             shutdown;

    public IdleConnectionEvictor(HttpClientConnectionManager connMgr, int httpClientIdleConnectionEvictionFrequency) {
        super();
        this.connMgr = connMgr;
        this.httpClientIdleConnectionEvictionFrequency = httpClientIdleConnectionEvictionFrequency;
        LOGGER.log(Level.INFO, "Started IdleConnectionEvictor for Apache Http Client");
    }

    @Override
    public void run() {

        try {
            Thread.sleep(30 * 1000L);
            boolean isLockAcquired = lock.tryLock(1, TimeUnit.SECONDS);
            if (!isLockAcquired)
                LOGGER.log(Level.ERROR, "Couldnt acquire lock in 1 second to recycle Stale Http connections");
            while (!shutdown && isLockAcquired && !Thread.currentThread().interrupted()) {


                Optional.ofNullable(connMgr).ifPresent(HttpClientConnectionManager::closeExpiredConnections);
                Optional.ofNullable(connMgr).ifPresent(connManager -> connManager
                        .closeIdleConnections(httpClientIdleConnectionEvictionFrequency, TimeUnit.SECONDS));
                Optional.ofNullable(connMgr).ifPresent(connManager -> LOGGER.log(Level.DEBUG,
                        "Closed ExpiredConnections and IdleConnections for Apache Http Client"));

            }

        } catch (InterruptedException ex) {
            LOGGER.log(Level.ERROR, "InterruptedException while recycling Stale Http connections ", ex);
        } finally {
            lock.unlock();
        }

    }

    public void shutdown() {
        shutdown = true;
        synchronized (this) {
            notifyAll();
        }
    }

}

RestService which invokes Http Requests

@Service
public class RestService{
    public <T> Response<T> call(HttpUriRequest request, ResponseHandler<T> responseHandler, long timeout) {
        Response<T> response;
        Optional<HttpResponse> optionalHttpResponse = null;
        CloseableHttpResponse httpResponse = null;
        try {
            try (CloseableHttpClient httpClient = getHttpClient()) {
                optionalHttpResponse = timeout == TIME_OUT_DISABLED ? execute(request, httpClient) : execute(request, httpClient, timeout);
                if (!optionalHttpResponse.isPresent())
                    throw new ClientMessageException("Empty/Null Response for " + request.getURI());
                httpResponse = (CloseableHttpResponse) optionalHttpResponse.get();
                HttpEntity entity = httpResponse.getEntity();
                try {
                    return new Response<>(httpResponse.getStatusLine().getStatusCode(), responseHandler.handleResponse(request, httpResponse, entity));
                } catch (Exception e) {
                    LOGGER.log(Level.ERROR, "Exception in Fetching Response from Server", e);
                    return new Response<>(httpResponse.getStatusLine().getStatusCode());
                } finally {
                    EntityUtils.consumeQuietly(entity);
                }

            }
        } catch (IOException e) {
            throw new ClientGeneralException(request, e);
        } finally {
            Optional.ofNullable(httpResponse).ifPresent(res -> {
                try {
                    res.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            });
            ((HttpRequestBase) request).releaseConnection();
        }
    }

    public Optional<HttpResponse> execute(HttpUriRequest request, Closeable httpClient) {

        if (!(httpClient instanceof CloseableHttpClient))
            throw new RuntimeException("UnSupported HttpClient Exception");
        CloseableHttpResponse httpResponse = null;
        try {
            CloseableHttpClient closeableHttpClient = (CloseableHttpClient) httpClient;
            httpResponse = closeableHttpClient.execute(request); //line 94
        } catch (ConnectionPoolTimeoutException e) {
            LOGGER.log(Level.ERROR,
                    "Connection pool is empty for request on uri: [" + request.getURI() + "]. Status code: ", e);
            throw new ResponseException("Connection pool is empty. " + e, request.getURI(), e);
        } catch (SocketTimeoutException | NoHttpResponseException e) {

            LOGGER.log(Level.ERROR, "Server on uri: [" + request.getURI() + "] is high loaded. Status code: ", e);
            throw new ResponseException("Remote server is high loaded. " + e, request.getURI(), e);
        } catch (ConnectTimeoutException e) {
            LOGGER.log(Level.ERROR, "HttpRequest is unable to establish a connection with the: [" + request.getURI()
                    + "] within the given period of time. Status code: ", e);
            throw new ResponseException(
                    "HttpRequest is unable to establish a connection within the given period of time. " + e,
                    request.getURI(), e);

        } catch (HttpHostConnectException e) {
            LOGGER.log(Level.ERROR, "Server on uri: [" + request.getURI() + "] is down. Status code: ", e);
            throw new ResponseException("Server is down. " + e, request.getURI(), e);
        } catch (ClientProtocolException e) {
            LOGGER.log(Level.ERROR, "URI: [" + request.getURI() + "]", e);
            throw new ResponseException(e.getMessage(), request.getURI(), e);
        } catch (IOException e) {
            LOGGER.log(Level.ERROR,
                    "Connection was aborted for request on uri: [" + request.getURI() + "]. Status code: ", e);
            throw new ResponseException("Connection was aborted. " + e, request.getURI(), e);
        }
        return Optional.ofNullable(httpResponse);

    }

    public Optional<HttpResponse> execute(HttpUriRequest request, Closeable httpClient, long timeOut) {
        Optional<HttpResponse> httpResponse;
        try {
            ExecutorService executorService = Executors.newCachedThreadPool();
            Future<Optional<HttpResponse>> future = executorService.submit(() -> execute(request, httpClient)); //line 129
            httpResponse = future.get(timeOut, TimeUnit.SECONDS);
            executorService.shutdown();
        } catch (InterruptedException | ExecutionException | TimeoutException e) {
            LOGGER.log(Level.ERROR, "Request execution error occured ", e);
            throw new ResponseException(e.getMessage(), request.getURI(), e);

        }
        return httpResponse;
    }
}

Below is the "Socket Closed" exception am getting randomly when am invoking https://reports.abc.com:8443/show.json?screenName=invoiceReport with 30 seconds timeOut, can somone please help if its the issue with the library or cant it be fixed with a configuration change?

http client request: POST https://reports.abc.com:8443/show.json?screenName=invoiceReport
log debug: o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
log debug: o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: JSESSIONID][value: F3 ... [expiry: null] match [(secure)reports.abc.com:8443/show.json]
log debug: o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: isDocumentIndexingInP ... [expiry: null] match [(secure)reports.abc.com:8443/show.json]
log debug: o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
log debug: o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {s}->http ... atacert.com:8443][total available: 56; route allocated: 66 of 400; total allocated: 67 of 400]
log debug: o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 134][route: { ... atacert.com:8443][total available: 56; route allocated: 67 of 400; total allocated: 68 of 400]
log debug: o.a.h.impl.execchain.MainClientExec - Opening connection {s}->https://reports.abc.com:8443
log debug: o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to reports.abc.com/10.10.10.10:8443
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Connecting socket to reports.abc.com/10.10.10.10:8443 with timeout 60000
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_A ... TH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
log info: o.a.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {s}->https://reports.abc.com:8443: Socket Closed
**log debug: o.a.http.impl.execchain.RetryExec - java.net.SocketException: Socket Closed**
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead()
    at java.net.SocketInputStream.read()
    at java.net.SocketInputStream.read()
    at sun.security.ssl.InputRecord.readFully()
    at sun.security.ssl.InputRecord.read()
    at sun.security.ssl.SSLSocketImpl.readRecord()
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake()
    at sun.security.ssl.SSLSocketImpl.startHandshake()
    at sun.security.ssl.SSLSocketImpl.startHandshake()
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at com.apps.http.rest.impl.RestService.execute(RestService.java:94)
    at com.apps.http.rest.impl.RestService.lambda$execute$1(RestService.java:129)
    at java.util.concurrent.FutureTask.run()
    at java.util.concurrent.ThreadPoolExecutor.runWorker()
    at java.util.concurrent.ThreadPoolExecutor$Worker.run()
    at java.lang.Thread.run()

log info: o.a.http.impl.execchain.RetryExec - Retrying request to {s}->https://reports.abc.com:8443
log debug: o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: default
log debug: o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: JSESSIONID][value: A2 ... [expiry: null] match [(secure)reports.abc.com:8443/show.json]
log debug: o.a.h.c.protocol.RequestAddCookies - Cookie [version: 0][name: isDocumentIndexingInP ... [expiry: null] match [(secure)reports.abc.com:8443/show.json]
log debug: o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
log debug: o.a.h.i.c.PoolingHttpClientConnectionManager - Connection request: [route: {s}->http ... atacert.com:8443][total available: 57; route allocated: 68 of 400; total allocated: 69 of 400]
log debug: o.a.h.i.c.PoolingHttpClientConnectionManager - Connection leased: [id: 143][route: { ... atacert.com:8443][total available: 57; route allocated: 69 of 400; total allocated: 70 of 400]
log debug: o.a.h.impl.execchain.MainClientExec - Opening connection {s}->https://reports.abc.com:8443
log debug: o.a.h.i.c.DefaultHttpClientConnectionOperator - Connecting to reports.abc.com/10.10.10.10:8443
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Connecting socket to reports.abc.com/10.10.10.10:8443 with timeout 60000
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_A ... TH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
log debug: o.a.h.c.s.SSLConnectionSocketFactory - Starting handshake
this auxiliary thread was still running when the transaction ended
log error: c.d.a.c.http.rest.impl.RestService - Request execution error occured
java.util.concurrent.TimeoutException
exception
log debug: o.a.h.impl.execchain.MainClientExec - Cancelling request execution
log debug: o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-134: Shutdown connection
log debug: o.a.h.impl.execchain.MainClientExec - Connection discarded
log debug: o.a.h.i.c.PoolingHttpClientConnectionManager - Connection released: [id: 134][route: { ... .abc.com:8443][total available: 57; route allocated: 68 of 400; total allocated: 69 of 400]

Answer 1

I don't know the reason for your exception, but I think that I may suggest some tool that may help you to diagnose the problem more effectively. Apache Http client is a great and widely accepted tool. However, in order to cover all parts of the Http protocol, it is a rather complex tool as well. So, sometimes it is useful to try some simplistic tool that may not provide such reach and complete functionality but is very simple in use and covers basic functionality that is enough in most of the cases. When I ran in similar problems I wrote my own Http client that is part of an Open Source library. I would suggest to try it instead of Apache Http client to see if the problem reproduces. If it doesn't - great, but if it does it might make it simpler to debug. The usage can be very simple:

HttpClient client = new HttpClient();
client.setConnectTimeout(timeOut, TimeUnit.MILISECONDS); //not required but may be useful
client.setReadTimeout(timeOut, TimeUnit.MILISECONDS);    //not required but may be useful
client.setContentType("...");
String content = client.sendHttpRequest(url, HttpClient.HttpMethod.GET);

BTW URL can be set for multiple uses with method public void setConnectionUrl(java.lang.String connectionUrl) for multiple re-uses. Then you send the request simple with method public java.lang.String sendHttpRequest(HttpClient.HttpMethod callMethod) .

Also, in the same library, there are some other utils that you might find useful. One of them a TimeUtils.sleepFor() method that does not require wrapping it in try-catch block. You can just write:
TimeUtils.sleepFor(30, TimeUnit.SECONDS);
and not worry about the InterruptedException . Another is Stacktrace filtering that makes it much easier to read the stacktrace. Anyway, the library is called MgntUtils and you can find it here on Maven Central and here on Github including source code and Javadoc. Javadoc separately could be viewed here And the article explaining about the library could be found here .

Answer 2

Based on information available I can suppose the cause of the issue is Apache HttpClient 4.5 negotiating a stronger TLS cipher and rejecting SSL protocol during SSL/TLS session negotiation compared to the behavior of version 4.0 which causes a server side error and connection termination by the server endpoint.

Answer 3

Due to the configured retry you do not see the actual root cause. The timeout is just the result of a problem that occured in one of the previous connection attemps. Either temporarily disable the retries or ensure that errors are logged before a retry is triggered.

Answer 4

It turned out that having requestRetry value greater than 0 is the root cause of the SocketException we have set the requestRetry = 0 and we were not getting any SocketException exceptions