數據流中的錯誤：io.grpc.StatusRuntimeException：不可用

Question

我正在嘗試建立一個從BigQuery讀取數據並將其發送到Pubsub主題的數據流作業。 我一直在相互依賴中掙扎，這是最新一期。

數據流作業正常啟動，但失敗，但出現以下異常：

com.google.cloud.pubsub.PubSubException：io.grpc.StatusRuntimeException：在com.google.cloud.pubsub.spi上的com.google.cloud.pubsub.spi.DefaultPubSubRpc $ 1.apply（DefaultPubSubRpc.java:174）不可用。 com.google.common.util.concurrent.Futures $ CatchingFuture.doFallback上的DefaultPubSubRpc $ 1.apply（DefaultPubSubRpc.java:168）位於com.google.common.util.concurrent.Futures $ CatchingFuture.doFallback（Futures.java:842） （Futures.java:834），位於com.google.common.util.concurrent.Futures $ AbstractCatchingFuture.run（Futures.java:789），位於com.google.common.util.concurrent.MoreExecutors $ DirectExecutor.execute（MoreExecutors.java ：456），位於com.google.com的com.google.common.util.concurrent.AbstractFuture.complete（AbstractFuture.java：753）的com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）上。 com.google.api.gax.grp上com.google.common.util.concurrent.SettableFuture.setException（SettableFuture.java:53）上的common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634） c.com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）的com.google.common.util.concurrent.MoreExecutors $ 1.onFailure（RetryingCallable.java:139） com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）上的com.google.common.util.concurrent.AbstractFuture.complete（AbstractFuture.java :)的DirectExecutor.execute（MoreExecutors.java:456）： 753），位於com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634），位於com.google.common.util.concurrent.SettableFuture.setException（SettableFuture.java:53），com.google.api .gax.grpc.ExceptionTransformingCallable $ 1.onFailure（ExceptionTransformingCallable.java:91）位於com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）位於com.google.common.util.concurrent.MoreExecutors com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）處com.google.common.util.concurrent.Ab的$ DirectExecutor.execute（MoreExecutors.java:456） com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）上的stractFuture.complete（AbstractFuture.java:753），io.grpc.stub.ClientCalls $ GrpcFuture.setException（ClientCalls.java:382）上的stractFuture.complete（AbstractFuture.java:753）在io.grpc.stub.ClientCalls $ UnaryStreamToFuture.onClose（ClientCalls.java:358）在io.grpc.internal.ClientCallImpl $ ClientStreamListenerImpl $ 3.runInContext（ClientCallImpl.java:462）在io.grpc.internal.ContextRunnable.run（ io.grpc.internal.SerializingExecutor $ TaskRunner.run（SerializingExecutor.java:154）處的ContextRunnable.java:54）java.util.concurrent.Executors $ RunnableAdapter.call（Executors.java:511）處的java.util.concurrent java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 201（ScheduledThreadPoolExecutor.java:180）處的.FutureTask.run（FutureTask.java:266）在java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor $ ScheduledFutureTask.run造成原因：com.google.api.gax.grpc.ApiException：io.grpc.StatusRuntimeException：UNAVAILABL E在com.google.api.gax.grpc.ExceptionTransformingCallable $ 1.onFailure（ExceptionTransformingCallable.java:91）在com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）在com.google。 common.util.concurrent.MoreExecutors $ DirectExecutor.execute（MoreExecutors.java:456）位於com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817），位於com.google.common.util.concurrent。 com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）上的AbstractFuture.complete（AbstractFuture.java:753）io.grpc.stub.ClientCalls $ GrpcFuture.setException（ClientCalls.java:382）上的AbstractFuture.setException（AbstractFuture.java:634）在io.grpc.stub.ClientCalls $ UnaryStreamToFuture.onClose（ClientCalls.java:358）在io.grpc.internal.ClientCallImpl $ ClientStreamListenerImpl $ 3.runInContext（ClientCallImpl.java:462）在io.grpc.internal.ContextRunnable.run（ ContextRunnable.java:54），位於io.grpc.internal.SerializingExecutor $ TaskRunner.run（SerializingExecutor.java:154），位於java.util.concurrent.Executors $ RunnableAdapter.c java.util.concurrent.FutureTask.run（FutureTask.java:266）處的java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 201（ScheduledThreadPoolExecutor.java:180）處的全部（Executors.java:511）。在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）處並發.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:293）在java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617在java.lang.Thread.run（Thread.java:745）導致原因：io.grpc.StatusRuntimeException：在io.grpc.Status.asRuntimeException（Status.java:431）處不可用...另有11個原因：javax。 net.ssl.SSLHandshakeException：在sun.security.ssl.SSLEngineImpl.kickstartHandshake（SSLEngineImpl.java）處的sun.security.ssl.Handshaker.activate（Handshaker.java:503）處沒有適當的協議（已禁用協議或密碼套件不適當） ：729），位於io.netty的sun.security.ssl.SSLEngineImpl.beginHandshake（SSLEngineImpl.java:756）上。 io.netty.handler.ssl.SslHandler.handshake（SslHandler.java:1363）上的handler.ssl.JdkSslEngine.beginHandshake（JdkSslEngine.java:147）在io.netty.handler.ssl.SslHandler.channelActive（SslHandler.java： 1405）在io.netty.channel.AbstractChannelHandlerContext.fireChannelActive（AbstractChannelHandlerContext.io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelActive（DefaultChannelHandlerInvoker.java:79）在io.netty.channel.AbstractChannelHandlerInvoker.java:48處：126），位於io.netty.channel.nio的io.netty.channel.nio.AbstractNioChannel $ AbstractNioUnsafe.fulfillConnectPromise（AbstractNioChannel.java:260）的io.netty.channel.DefaultChannelPipeline.fireChannelActive（DefaultChannelPipeline.java:884）。 io.netty.channel.nio.NioEventLoop.processSelectedKey（NioEventLoop.java:527）處的io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized（NioEventLoop.java:467）處的AbstractNioChannel $ AbstractNioUnsafe.finishConnect（AbstractNioChannel.java:290）在 io.netty.channel.nio.io.netty.channel.nio.NioEventLoop.processSelectedKeys（NioEventLoop.java:381）在io.netty.util.concurrent.SingleThreadEventExecutor $ 5.run上的io.netty.channel.nio.NioEventLoop.run（NioEventLoop.java:353） （SingleThreadEventExecutor.java:742）...還有3個

我懷疑這與所使用的netty / ssl庫有關，但我不知道正確的做法是什么。 我花了很多時間試圖從生成的有陰影的jar中排除某些庫的某些版本，在這些罐中類重疊了，但是最終導致找不到類異常。

這些是我的依賴項：

<dependencies>
  <dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>1.8.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
    <version>0.3.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.0.0-beta-1</version>
  </dependency>
</dependencies>

這是依賴關系樹：

[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ dataflow-bulk-load ---
[INFO] com.google.lindsaysmith.titan:dataflow-bulk-load:jar:0.0.1-SNAPSHOT
[INFO] +- com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:1.8.0:compile
[INFO] |  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev36-1.22.0:compile
[INFO] |  +- io.grpc:grpc-all:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-auth:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-netty:jar:0.13.1:compile
[INFO] |  |  |  \- io.netty:netty-codec-http2:jar:4.1.0.CR1:compile
[INFO] |  |  |     \- io.netty:netty-codec-http:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-core:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-okhttp:jar:0.13.1:compile
[INFO] |  |  |  +- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] |  |  |  \- com.squareup.okhttp:okhttp:jar:2.5.0:compile
[INFO] |  |  +- io.grpc:grpc-protobuf-nano:jar:0.13.1:compile
[INFO] |  |  |  \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
[INFO] |  |  \- io.grpc:grpc-stub:jar:0.13.1:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-common:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.0.CR1:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.0.CR1:compile
[INFO] |  +- com.google.api.grpc:grpc-pubsub-v1:jar:0.0.2:compile
[INFO] |  |  \- com.google.api.grpc:grpc-core-proto:jar:0.0.3:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.4.0:compile
[INFO] |  |  \- com.google.auth:google-auth-library-credentials:jar:0.4.0:compile
[INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.3.0:compile
[INFO] |  |  \- com.google.guava:guava:jar:19.0:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.22.0:compile
[INFO] |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client-java6:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-4.0.0:compile
[INFO] |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  +- com.google.api-client:google-api-client-java6:jar:1.22.0:compile
[INFO] |  |  \- com.google.api-client:google-api-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.1.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-protobuf:jar:1.22.0:compile
[INFO] |  |  \- com.google.http-client:google-http-client-jackson:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.0.1:compile
[INFO] |  +- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.14:compile
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.1:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.9:compile
[INFO] |  \- joda-time:joda-time:jar:2.4:compile
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.3.0:compile
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork17:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:0.3.0:compile
[INFO] |  |  +- com.google.api-client:google-api-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  +- com.google.oauth-client:google-oauth-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  |  +- com.google.oauth-client:google-oauth-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  |  \- com.google.http-client:google-http-client-jdo:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |  |  +- com.google.api-client:google-api-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.jdo:jdo2-api:jar:2.3-eb:compile
[INFO] |  |  |  |     \- javax.transaction:transaction-api:jar:1.1:compile
[INFO] |  |  |  \- com.google.http-client:google-http-client-appengine:jar:1.21.0:compile
[INFO] |  |  +- org.json:json:jar:20151123:compile
[INFO] |  |  +- com.google.api:gax:jar:0.0.16:compile
[INFO] |  |  |  \- com.google.inject:guice:jar:4.0:compile
[INFO] |  |  |     +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |     \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.0.7:compile
[INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.0.7:compile
[INFO] |  \- com.google.auto.value:auto-value:jar:1.1:compile
[INFO] \- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.749s
[INFO] Finished at: Sun Dec 04 16:50:41 CET 2016
[INFO] Final Memory: 16M/972M
[INFO] ------------------------------------------------------------------------

Answer 1

當我嘗試使用PubSub庫為DataFlow中的每個已處理項目手動發送消息時，就會發生此問題。

取而代之的是，我轉而使用DataFlow的PubSub IO Sink類來發送消息，並且這種方法完美無缺。

文件：

https://cloud.google.com/dataflow/model/pubsub-io#writing-with-pubsubio