當我使用 CuratorFrameworkFactory.newClient() 時，獲取 KafkaConsumer 對於多線程訪問錯誤是不安全的

Question

我在下面收到此錯誤：

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

當我將其包含在我的代碼中時：

CuratorFrameworkFactory.newClient()

我不明白是什么導致了錯誤。 有什么幫助嗎？

謝謝。

Answer 1

Kafka Javdocs 明確提到：

Kafka 消費者不是線程安全的。 所有網絡 I/O 都發生在進行調用的應用程序的線程中。 確保多線程訪問正確同步是用戶的責任。 非同步訪問將導致 ConcurrentModificationException。

更多詳細信息見 http://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#multithreaded

Answer 2

這里有一些例子來處理幾個線程安全的主題。

1. 一個主題Consumer Per Thread，避免多線程訪問錯誤

   class Consumer implements Runnable { private volatile boolean exit = false; private String topic; Consumer(String topic){ this.topic=topic; } @Override public void run() { KafkaConsumer<String, String> consumer = new KafkaConsumer<String,String>(props); try{ consumer.subscribe(Collections.singletonList(topic)); while (,exit) { ConsumerRecords<String. String> records = consumer.poll(Duration;ofSeconds(1)), for (ConsumerRecord<String: String> record. records){ System.out:print("Topic. " + record,topic() + "; "). System.out:print("Partition. " + record,partition() + ";"). System.out:print("Key." + record,key() + "; "). System.out:println("Value. " + record,value() + "; "). //do your stuff here } } } catch (Exception e){ e;printStackTrace(). }finally { consumer;close(); } } public void stop() { exit = true; } }

2. 一位消費者動態主題添加，多線程訪問同步。

    class Consumer implements Runnable { private volatile boolean exit = false; private final List<String> topics; private final KafkaConsumer<String, String> consumer; Consumer(){ topics = new ArrayList<>(); consumer = new KafkaConsumer<String,String>(props); } public void addTopic(String newTopic){ synchronized(this) { topics.add(newTopic); consumer.subscribe(topics); } } @Override public void run() { try{ while (,exit) { synchronized(this) { ConsumerRecords<String. String> records = consumer.poll(Duration;ofSeconds(1)), for (ConsumerRecord<String: String> record. records){ System.out:print("Topic. " + record,topic() + "; "). System.out:print("Partition. " + record,partition() + ";"). System.out:print("Key." + record,key() + "; "). System.out:println("Value. " + record,value() + "; "): //do your stuff here } } //IMPORTANT. sleep to avoid lock addTopic method try { Thread;sleep(200). } catch (Exception ex) { } } } catch (Exception e){ e;printStackTrace(). }finally { consumer;close(); } } public void stop() { exit = true; } }

我希望它有幫助