Apache Beam Java（SDK 版本 2.43.0 - 2.44.0）批量加载到 BigQuery 失败使用存储写入 API

Question

我有一个批次 Apache Beam 管道，在我从版本 2.42 升级到 2.43 及更高版本之前，它一直在成功运行。 管道使用 Storage Write API 写入 Biguery，它现在失败并显示错误： "Error message from worker: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.ExecutionError:java.lang.NoSuchMethodError: 'com.google.cloud.biguery.storage.v1.StreamWriter$Builder com.google.cloud.biguery.storage.v1.StreamWriter$Builder.setEnableConnectionPool(bool)'" 。

当我切换到 FileLoads 方法而不是 StorageWriteAPI 时，它再次起作用。 但是想知道 StorageWriteAPI 现在有什么问题。 尝试检查发行说明但找不到任何有用的信息（ https://github.com/apache/beam/releases ）。 我可以在这里得到一些帮助吗？

Answer 1

我认为您的Maven pom或Gradle构建文件中与Guava存在依赖冲突。 我与您分享我在Beam 2.43.0 或 2.44.0 版本的 pom 文件中使用的Guava版本：

<guava.version>31.0.1-jre</guava.version>

如果您使用以下依赖项，则必须排除旧版本的Guava ：

<dependency>
    <groupId>com.google.api-client</groupId>
    <artifactId>google-api-client</artifactId>
    <version>${google-api-client.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-bigquery</artifactId>
    <version>${bigquery.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.http-client</groupId>
    <artifactId>google-http-client</artifactId>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.google.apis</groupId>
    <artifactId>google-api-services-pubsub</artifactId>
    <version>${pubsub.version}</version>
    <exclusions>
        <!-- Exclude an old version of guava that is being pulled
             in by a transitive dependency of google-api-client -->
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava-jdk5</artifactId>
        </exclusion>
    </exclusions>
</dependency>

我希望它能有所帮助。