java.lang.IllegalStateException：連接池關閉於

Question

從 S3 下載文件時失敗並給我這個異常

java.lang.IllegalStateException：連接池關閉

在 org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws .http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy70.requestConnection(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java :175) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client. CloseableHttpClient.execute（CloseableHttpClient.java:82）在 org.apache.http.impl.client.Closea bleHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer (AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws .http.AmazonHttpCli ent$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com .amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249) at com.amazonaws.services.s3.transfer.TransferManager.doDownload (TransferManager.java:1053) at com.amazonaws.services.s3.transfer.TransferManager.doDownload(TransferManager.java:1007) at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:845) at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:801) 在 com.capitalone.homeloans.imaging.common.s3communicato r.AsyncS3Manager.lambda$download$0(AsyncS3Manager.java:82) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142 ) 在 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 在 java.lang.Thread.run(Thread.java:745)

Answer 1

如果您正在匯集連接，請不要在請求后關閉您的客戶端。

也就是說，你可能正在做這樣的事情：

PoolingHttpClientConnectionManager pool = new PoolingHttpClientConnectionManager();
...
CloseableHttpClient httpclient = HttpClients.custom()
     .setConnectionManager(pool)
     .build();

try { // try-with-resources
    HttpGet httpget = new HttpGet(url.toURI());
    try (CloseableHttpResponse response = httpclient.execute(httpget);
             InputStream fis = response.getEntity().getContent();
            ReadableByteChannel channel = Channels.newChannel(fis)) {
             // ... get data ...
     } finally {
         httpclient.close(); <====== !!
     }
} catch (IOException | URISyntaxException e) {
    // exception handling ...
}

該httpclient.close()導致您的下一個池連接失敗。

---

^{摘自： https : //stackoverflow.com/a/59033548/1329340}

Answer 2

在 databricks 運行時環境中，我成功地使用了本文中的解決方案，並結合了配置鍵前綴的小變化：

spark.hadoop.fs.s3a.connection.maximum 3

原來的文章解決方案說使用：

fs.s3a.connection.maximum

https://kb.databricks.com/en_US/jobs/job-fails-connection-pool