ProtoOperationTransformers的'Java.lang.NoClassDefFoundError`在运行时发生

Question

I am using Apache Beam for Java and are using Cloud DLP API with Cloud Dataflow. Job starts, but it gets an error at run time.

I think that it is a problem of the combination of the Version of the Library of gRPC running on DataLink with the Client Library of DLP API, but I do not know which version to specify.

dependency

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-dlp</artifactId>
  <version>0.33.0-beta</version>
</dependency>

<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax</artifactId>
  <version>1.16.0</version>
</dependency>
<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax-grpc</artifactId>
  <version>0.20.0</version>
</dependency>
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>3.2.0</version>
</dependency>

Error

java.lang.RuntimeException: java.lang.NoClassDefFoundError: com/google/api/gax/grpc/ProtoOperationTransformers$ResponseTransformer
    at org.sinmetal.mlapi.DataLossPreventionFn.processElement(DataLossPreventionFn.java:52)
Caused by: java.lang.NoClassDefFoundError: com/google/api/gax/grpc/ProtoOperationTransformers$ResponseTransformer
    at com.google.cloud.dlp.v2beta1.DlpServiceSettings$Builder.initDefaults(DlpServiceSettings.java:425)
    at com.google.cloud.dlp.v2beta1.DlpServiceSettings$Builder.<init>(DlpServiceSettings.java:363)
    at com.google.cloud.dlp.v2beta1.DlpServiceSettings$Builder.createDefault(DlpServiceSettings.java:367)
    at com.google.cloud.dlp.v2beta1.DlpServiceSettings$Builder.access$000(DlpServiceSettings.java:264)
    at com.google.cloud.dlp.v2beta1.DlpServiceSettings.newBuilder(DlpServiceSettings.java:233)
    at com.google.cloud.dlp.v2beta1.DlpServiceClient.create(DlpServiceClient.java:149)
    at org.sinmetal.mlapi.DataLossPreventionFn.processElement(DataLossPreventionFn.java:26)
    at org.sinmetal.mlapi.DataLossPreventionFn$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:324)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:272)
    at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
    at org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
    at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
    at org.apache.beam.sdk.io.gcp.bigquery.PassThroughThenCleanup$IdentityFn.processElement(PassThroughThenCleanup.java:83)
    at org.apache.beam.sdk.io.gcp.bigquery.PassThroughThenCleanup$IdentityFn$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
    at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:324)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:187)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:148)
    at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:68)
    at com.google.cloud.dataflow.worker.DataflowWorker.executeWork(DataflowWorker.java:330)
    at com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:302)
    at com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:251)
    at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:135)
    at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:115)
    at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:102)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Answer 1

You are using very old version of gax-grpc , in older version of library ProtoOperationTransformers doesn't exists. You need to update your pom.xml to point to latest version

<dependency>
   <groupId>com.google.api</groupId>
   <artifactId>gax-grpc</artifactId>
   <version>1.16.0</version>
</dependency>

Note : Then, update your all related libraries to latest version.

Answer 2

I was able to run Cloud DLP by specifying the library as follows.

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-dlp</artifactId>
  <version>0.33.0-beta</version>
</dependency>

<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax</artifactId>
  <version>1.16.0</version>
</dependency>
<dependency>
  <groupId>com.google.api</groupId>
  <artifactId>gax-grpc</artifactId>
  <version>1.16.0</version>
</dependency>
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>3.5.1</version>
</dependency>