Solr 8.4 Getting Async exception during distributed update: java.io.IOException: Broken pipe when trying to post a document

Question

為什么我看到錯誤：

Async exception during distributed update: java.io.IOException: Broken pipe when trying to post a document to solr

• Solr 版本：8.4.1
• 動物園管理員：3.4.14
• 開放JDK 11；
• 2 solr 節點 + 1 個 zookeeper（在 solr 節點之一中托管 zookeeper）

在 solr 和TLS1.2上使用基本身份驗證。

嘗試發布到 solr 節點時，我沒有看到任何錯誤，該節點是領導者並且它一直在工作； 只有當我發布到拋出以下錯誤的非領導節點時，這才是問題。

java.io.IOException: java.io.IOException: Broken pipe
    at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
    at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
    at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
    at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
    at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
    at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:172)
    at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:103)
    at org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
    at org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:339)
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:236)
    at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:181)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$142/0000000000000000.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
    Suppressed: java.io.IOException: java.io.IOException: Broken pipe
        at org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
        at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
        at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
        at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
        at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
        at org.apache.solr.common.util.JavaBinCodec.close(JavaBinCodec.java:1269)
        at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:104)
        ... 10 more
    Caused by: java.io.IOException: Broken pipe
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:927)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:393)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:380)
        at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:247)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
        at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:755)
        at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:734)
        at org.eclipse.jetty.http2.client.HTTP2ClientConnectionFactory$HTTP2ClientConnection.onOpen(HTTP2ClientConnectionFactory.java:130)
        at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:441)
        at org.eclipse.jetty.io.NegotiatingClientConnection.replaceConnection(NegotiatingClientConnection.java:115)
        at org.eclipse.jetty.io.NegotiatingClientConnection.onFillable(NegotiatingClientConnection.java:85)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)

Answer 1

• 由於服務器等待時間過長，客戶端斷開連接。 較高的超時可能會導致性能問題。

• 另一個原因可能是操作系統緩存。 檢查磁盤空間並嘗試增加磁盤空間。

Answer 2

當我們遇到這個問題時，是因為我們運行了錯誤的 JDK。 我們已經安裝了 OpenJDK 16，盡管更新到了較新的版本，但 Solr 和 Zookeeper 仍在使用它。 當領導者與追隨者通信時，存在一些不兼容導致損壞的 pipe 錯誤。

檢查您正在運行的 OpenJDK 版本，並確保 Solr 和 Zookeeper 使用的是預期版本。