Flink集群构建Kafka Consumer连接MSK失败

Question

我正在尝试使用 Flink-kafka-connector 将我的 Flink 应用程序连接到 AWS 中的 MSK。 Flink 应用在 EC2 实例上运行。

flink.version：1.9.2 pom.xml：

   <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.11</artifactId>
        <version>${flink.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>software.amazon.msk</groupId>
        <artifactId>aws-msk-iam-auth</artifactId>
        <version>1.1.4</version>
    </dependency>
   <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-kafka_2.11</artifactId>
        <version>${flink.version}</version>
        <scope>compile</scope>
    </dependency>

Flink TaskManager 日志中的异常：

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
    at org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:58)
    at org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94)
    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:505)
    at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
    at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:552)
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:416)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.auth.AWSCredentialsProviderChain
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:112)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:726)
    ... 12 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.auth.AWSCredentialsProviderChain
    at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getDefaultProvider(MSKCredentialProvider.java:123)
    at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.<init>(MSKCredentialProvider.java:104)
    at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.<init>(MSKCredentialProvider.java:94)
    at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.<init>(MSKCredentialProvider.java:90)
    at software.amazon.msk.auth.iam.IAMClientCallbackHandler.lambda$configure$1(IAMClientCallbackHandler.java:54)
    at java.util.Optional.map(Optional.java:215)
    at software.amazon.msk.auth.iam.IAMClientCallbackHandler.configure(IAMClientCallbackHandler.java:54)
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:130)
    ... 16 more

在构建过程中，我可以清楚地看到 maven shade 插件在 uber jar 中包含了 AWS 所需的所有库：

[INFO] Including software.amazon.msk:aws-msk-iam-auth:jar:1.1.4 in the shaded jar.
[INFO] Including com.amazonaws:aws-java-sdk-core:jar:1.11.986 in the shaded jar.
[INFO] Including software.amazon.ion:ion-java:jar:1.0.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.7 in the shaded jar.
[INFO] Including com.amazonaws:aws-java-sdk-sts:jar:1.11.986 in the shaded jar.
[INFO] Including com.amazonaws:jmespath-java:jar:1.11.986 in the shaded jar.
[INFO] Including software.amazon.awssdk:auth:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:annotations:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:utils:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:sdk-core:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:regions:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:profiles:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:http-client-spi:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:json-utils:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:third-party-jackson-core:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.eventstream:eventstream:jar:1.0.1 in the shaded jar.
[INFO] Including software.amazon.awssdk:sso:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:protocol-core:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:aws-json-protocol:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:aws-core:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:metrics-spi:jar:2.17.192 in the shaded jar.
[INFO] Including software.amazon.awssdk:netty-nio-client:jar:2.17.192 in the shaded jar.

我的 Kafka 消费者配置：

org.apache.kafka.clients.consumer.ConsumerConfig              - ConsumerConfig values: 
    auto.commit.interval.ms = 5000
    auto.offset.reset = latest
    bootstrap.servers = [***]
    check.crcs = true
    client.dns.lookup = default
    client.id = 
    connections.max.idle.ms = 540000
    default.api.timeout.ms = 60000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = msk-gps-testing-FlinkRealTimeGPS
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 500
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = AWS_MSK_IAM
    security.protocol = SASL_SSL
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /opt/certs/truststore.jks
    ssl.truststore.password = [hidden]
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

我还检查了 flink 集群上的凭据，这是我得到的：

ubuntu@flink1-10:~$ aws configure list

      Name                    Value             Type    Location

      ----                    -----             ----    --------

   profile                <not set>             None    None

access_key     ********************         iam-role    

secret_key     ********************         iam-role    

    region                <not set>             None    None

我已经阅读了 AWS 提供的所有这些文档： https ://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html

我已经尝试了很多方法来解决它，但我找不到任何答案。

Answer 1

Flink 1.9.2 不支持 MSK 连接的 IAM 身份验证。 为了解决这个问题，我不得不将 Flink 版本升级到 1.13.2，然后将以下 jars 上传到 Flink 服务器类路径：

AWS lib aws-msk-iam-auth-1.1.4-all.jar 和 Kafka lib kafka-clients-2.4.1.jar 放入 Flink 节点内的 /opt/flink/lib 文件夹中。

在此之后，我重新启动了 Flink 服务，然后作业成功运行。