突然报错java.lang.IllegalStateException: Request cannot be executed; I/O反应器状态：STOPPED

Question

I'm using httpcore-nio-4.4.5.jar. I'm using the elasticsearch RestHighLevelClient to interact with our elasticsearch servers. This all works fine except for some time we get I/O reactor stopped error out of a sudden.

Everything seems fine on ES side. No strange behaviour.

That's how i'm initializing my ES client.

public synchronized RestHighLevelClient getHighLevelClient() throws ManagerException {
        if (highLevelClient != null) {
            return highLevelClient;
        }

        Map<String, Integer> map = getEsServers(esAddresses);

        HttpHost[] hosts = new HttpHost[map.size()];

        int i = 0;

        for (Map.Entry<String, Integer> entry : map.entrySet()) {
            hosts[i++] = new HttpHost(entry.getKey(), entry.getValue(), "http");
            LOGGER.info(entry.getKey() + " " + entry.getValue());
        }

        RestClientBuilder restClientBuilder = RestClient.builder(hosts);
        highLevelClient = customizeHttpClient(restClientBuilder);
        return highLevelClient;
    }

public RestHighLevelClient customizeHttpClient(RestClientBuilder restClientBuilder) {
        Header[] defaultHeaders = new Header[2];
        defaultHeaders[0] = new BasicHeader("Authorization", "Basic YTph");
        defaultHeaders[1] = new BasicHeader("Accept", "application/json");

        restClientBuilder.setDefaultHeaders(defaultHeaders);

        restClientBuilder.setMaxRetryTimeoutMillis(MAX_RETRY_TIMEOUT_MILLIS);

        restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                .setConnectTimeout(CONNECT_TIMEOUT_MILLIS)
                .setSocketTimeout(SOCKET_TIMEOUT_MILLIS)
                .setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT_MILLIS));

        restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                .setMaxConnPerRoute(MAX_CONN_PER_ROUTE)
                .setMaxConnTotal(MAX_CONN_TOTAL));

        return new RestHighLevelClient(restClientBuilder);
    }

So basically first I get the following stacktrace

java.lang.IllegalStateException: I/O reactor has been shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34) 
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:224) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:434) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:276) 
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:266) 
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:363) 
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:125) 
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515) 
        at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400) 

and after that no Timeout just the following exception continuously until i restart my servers.

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
    at org.apache.http.util.Asserts.check(Asserts.java:46)  
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)  
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515)  
    at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400)  ```

Answer 1

I was facing the same issue, and it turned out I was not closing the HighLevelRestClient.

Refer to HenningAndersen answer at https://discuss.elastic.co/t/request-cannot-be-executed-io-reactor-status-stopped/195438/4 :

This sounds similar to https://github.com/elastic/elasticsearch/issues/45115 141. The problem in that issue is caused by throwing an exception in an onFailure method in the client application code. This causes the connection to be closed and subsequent requests will fail with the error reported here.

I believe this also affects the high level client if using any of the methods reporting response/failure back through an ActionListener (I think all of them have suffix Async in their method names).

The workaround is to ensure that no exceptions are thrown out of your onFailure methods.

Answer 2

Change the condition of getHighLevelClient() to if (highLevelClient.= null && highLevelClient.getLowLevelClient().isRunning())

which means if the low level client (RestClient) is not running, you should renew the client, that works for me.