[英]SSLHandshakeException when running Apache Beam Pipeline in Dataflow
[英]GCP Dataflow - SSLHandshakeException
在運行大型(> 1000 cpus)和中型(100-1000 cpus)數據流作業時遇到以下問題:
異常:“ javax.net.ssl.SSLHandshakeException:握手期間遠程主機關閉了連接
該錯誤不是致命的,但是一旦開始,它大約每30秒重現一次。 顯示此錯誤的作業永遠不會完成(我已經等待了預期運行時間的4倍以上),並且產生的結果非常有限(不到預期輸出的4%)。 產量有限時,會在工作開始時收到,之后不再生產。
我同時使用BigQueryIO和JdbcIO Apache梁源和接收器。
重要的是要注意,我的工作在6月初可以正常工作,但是自7月初以來已經開始顯示此錯誤。
我在Google的企業支持中有一個公開的案例,但是只能說結果不理想。 Google唯一感興趣的一點是,“如果工作人員擴大規模並大量訪問Cloud Storage,則可能會發生錯誤”。
但是,該語句沒有附加解決方案。 這是記錄在日志中的完整錯誤的示例:
exception: "javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call(ResilientOperation.java:166)
at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.getMetadata(GoogleCloudStorageReadChannel.java:573)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStreamAndSetMetadata(GoogleCloudStorageReadChannel.java:645)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:560)
at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:289)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.InputStream.read(InputStream.java:101)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
at org.apache.beam.sdk.util.VarInt.decodeLong(VarInt.java:79)
at org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:63)
at org.apache.beam.runners.dataflow.internal.IsmFormat$KeyPrefixCoder.decode(IsmFormat.java:709)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader.readKey(IsmReader.java:1001)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$2000(IsmReader.java:79)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.advance(IsmReader.java:953)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.start(IsmReader.java:943)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:581)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:570)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:555)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader.fetch(IsmReader.java:606)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader.getBlock(IsmReader.java:771)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$1000(IsmReader.java:79)
at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmPrefixReaderIterator.get(IsmReader.java:642)
at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.getUsingLong(IsmSideInputReader.java:679)
at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.access$1300(IsmSideInputReader.java:625)
at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators$ListIteratorOverReaderIterators.next(IsmSideInputReader.java:720)
at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
at com.application.strategy.simulator.MainStrategySimulator$1.processElement(MainStrategySimulator.java:224)
at com.application.strategy.simulator.MainStrategySimulator$1$auxiliary$4N23tth9.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:199)
at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:157)
at com.google.cloud.dataflow.worker.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:244)
at com.google.cloud.dataflow.worker.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at com.google.cloud.dataflow.worker.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:198)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72)
at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:336)
at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:295)
at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:242)
at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:123)
at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:103)
at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:90)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 69 more
在聯機查看此問題的其他實例之后,該錯誤似乎僅在使用Java 7時發生。這是您的Java版本嗎?
如果是,我建議嘗試Java 8,看看是否可以解決該問題。 讓我知道我們能否進一步提供幫助!
相關的github問題:
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.