简体繁体 English

KafkaConsumer 跨线程可见性

[英]KafkaConsumer visibility across threads

原文 2021-10-26 08:11:13 4 1 java/ apache-kafka/ kafka-consumer-api

I have an application which has two threads running.我有一个运行两个线程的应用程序。 These threads share a KafkaConsumer instance between them.这些线程在它们之间共享一个 KafkaConsumer 实例。

KafkaConsumer is created by disabling the auto commit config. KafkaConsumer 是通过禁用自动提交配置创建的。 One thread will assign topic-partitions to the KafkaConsumer instance based on some external input.一个线程将根据一些外部输入将主题分区分配给 KafkaConsumer 实例。 The other thread constantly polls the topic, process the records and commit the offsets at the end.另一个线程不断轮询主题，处理记录并在最后提交偏移量。

Now, let's say the topic-partitions assignment was changed in Thread-1.现在，假设主题分区分配在 Thread-1 中已更改。 What can we do to make sure that Thread-2 can see these changes?我们可以做些什么来确保 Thread-2 可以看到这些变化？ ie Thread-2's KafkaConsumer will know that topic-partitions have been updated in Thread-1.即 Thread-2 的 KafkaConsumer 将知道主题分区已在 Thread-1 中更新。

Does defining the variable as volatile solve this issue?将变量定义为 volatile 能解决这个问题吗？ Or should we do anything more?或者我们应该做些什么？ Please provide your explanantions with code examples as I am new to multi-threading in Java.请提供您的解释和代码示例，因为我是 Java 多线程的新手。

1 个解决方案

One thread will assign topic-partitions to the KafkaConsumer instance based on some external input.一个线程将根据一些外部输入将主题分区分配给 KafkaConsumer 实例。 The other thread constantly polls the topic, process the records and commit the offsets at the end.另一个线程不断轮询主题，处理记录并在最后提交偏移量。

According to the javadoc (Section Multi-threaded Processing ):根据javadoc（多线程处理部分）：

The Kafka consumer is NOT thread-safe. Kafka 消费者不是线程安全的。

With that I'm not so sure you can change partition assignment in one thread and see the changes in the other.有了这个，我不太确定您可以在一个线程中更改分区分配并查看另一个线程中的更改。