Java 版本与 Protobuf 冲突

Question

我正在创建一个 GRPC 端点，它将接受数据并将数据传递到 MapR Streams（Kafka 端口）。 我已经创建了 GPRC 端点，并且在将数据转储到控制台时工作正常。 当我添加代码以将消息推送到 MapR/Kafka 时，它编译得很好，但是当我运行它时，我得到一个与 protobuf 库版本相关的错误。 MapR 是针对 2.5.0 编译的，我在项目中使用的是 3.5.1。 那么有没有办法让我自己使用最新版本，同时让 MapR 库使用旧版本？ 请注意，过去我已将 protobuf 版本回滚到 2.5.x 并由 ptoto 文件编辑为版本 2 文件。 这是可能的，但很痛苦，因为人们通常会向我发送 v3 文件。

我在 maven 中的依赖项是：

    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>3.5.1</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-stub -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.30.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-protobuf -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>1.31.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-netty -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty</artifactId>
        <version>1.31.0</version>
    </dependency>

    <dependency>
        <groupId>com.mapr.streams</groupId>
        <artifactId>mapr-streams</artifactId>
        <version>6.1.0-mapr</version>
    </dependency>

错误是：

[d336599@mapredge01 ~]$ java -jar TLiveEndPoint-jar-with-dependencies.jar 
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Aug 07, 2020 3:26:39 PM HelloWorldClient greet
INFO: Will try to greet world ...
Exception in thread "main" java.lang.RuntimeException: Error occurred while instantiating com.mapr.streams.impl.producer.MarlinProducerV10.
==> java.lang.VerifyError: class com.mapr.fs.proto.Marlinserver$MarlinInternalDefaults overrides final method getParserForType.()Lcom/google/protobuf/Parser;.
        at org.apache.kafka.clients.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:42)
        at org.apache.kafka.clients.producer.KafkaProducer.initializeProducer(KafkaProducer.java:419)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:914)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:801)
        at MapRProducer.sendMessage(MapRProducer.java:16)
        at HelloWorldClient.greet(HelloWorldClient.java:51)
        at HelloWorldClient.main(HelloWorldClient.java:96)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.kafka.clients.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:39)
        ... 6 more
Caused by: java.lang.VerifyError: class com.mapr.fs.proto.Marlinserver$MarlinInternalDefaults overrides final method getParserForType.()Lcom/google/protobuf/Parser;
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.mapr.streams.impl.producer.MarlinProducerImpl.<init>(MarlinProducerImpl.java:73)
        at com.mapr.streams.impl.producer.MarlinProducerImplV10.<init>(MarlinProducerImplV10.java:27)
        at com.mapr.streams.impl.producer.MarlinProducerV10.<init>(MarlinProducerV10.java:10)
        ... 11 more

Answer 1

现在，我们将 MapR 库及其旧的 protobuf 依赖项以及我们的一些便捷方法隐藏到我们自己的“表访问”JAR 中。

您必须包含所有需要该 protobuf 依赖项的依赖项，因为这将重新定位旧 protobuf 类的包，并重写依赖项以引用那些重新定位的名称。

使用 Maven 和 maven-shade-plugin，最终看起来像这样：

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-shade-plugin</artifactId>
          <version>${maven-shade-plugin.version}</version>
          <executions>
            <execution>
              <id>shade-mapr-protobuf</id>
              <phase>package</phase>
              <goals>
                <goal>shade</goal>
              </goals>
              <configuration>
                <relocations>
                  <relocation>
                    <pattern>com.google.protobuf</pattern>
                    <shadedPattern>com.example.mapr.com.google.protobuf</shadedPattern>
                  </relocation>
                </relocations>
                <artifactSet>
                  <includes>
                    <include>com.mapr.fs:mapr-hbase</include>
                    <include>com.mapr.db:*</include>
                    <include>com.mapr.hadoop:maprfs</include>
                    <include>org.apache.hbase:*</include>
                    <include>org.apache.hadoop:*</include>
                    <include>com.google.protobuf:*</include>
                  </includes>
                </artifactSet>
                <filters>
                  <filter>
                    <artifact>*:*</artifact>
                    <excludes>
                      <exclude>org/apache/hadoop/yarn/util/package-info.class</exclude>
                      <exclude>org/apache/hadoop/yarn/factories/package-info.class</exclude>
                      <exclude>org/apache/hadoop/yarn/factory/providers/package-info.class</exclude>
                    </excludes>
                  </filter>
                </filters>
                <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
                <useBaseVersion>true</useBaseVersion>
              </configuration>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
...
  </build>