java.lang.IllegalStateException：连接池关闭于

Question

While downloading file from S3 it is failing and giving me this exception

java.lang.IllegalStateException: Connection pool shut down

at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy70.requestConnection(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.Closea bleHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpCli ent$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249) at com.amazonaws.services.s3.transfer.TransferManager.doDownload(TransferManager.java:1053) at com.amazonaws.services.s3.transfer.TransferManager.doDownload(TransferManager.java:1007) at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:845) at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:801) at com.capitalone.homeloans.imaging.common.s3communicato r.AsyncS3Manager.lambda$download$0(AsyncS3Manager.java:82) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Answer 1

Do not close your client after a request if you are pooling the connections.

That is, you are probably doing something like this:

PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager();
...
CloseableHttpClient httpclient = HttpClients.custom()
     .setConnectionManager(pool)
     .build();

try { // try-with-resources
    HttpGet httpget = new HttpGet(url.toURI());
    try (CloseableHttpResponse response = httpclient.execute(httpget);
             InputStream fis = response.getEntity().getContent();
            ReadableByteChannel channel = Channels.newChannel(fis)) {
             // ... get data ...
     } finally {
         httpclient.close(); <====== !!
     }
} catch (IOException | URISyntaxException e) {
    // exception handling ...
}

That httpclient.close() is causing your next pooled connection to fail.

---

^{Taken from: https://stackoverflow.com/a/59033548/1329340}

Answer 2

in the databricks runtime environment, i found success using the solution from this article mixed with a small variation on the prefix for the configuration key:

spark.hadoop.fs.s3a.connection.maximum 3

the original article solution said to use:

fs.s3a.connection.maximum

https://kb.databricks.com/en_US/jobs/job-fails-connection-pool