[英]Concurrently Consume Multiple topics as a kafka consumer
我正在關注這個 url https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example 中關於並發消費 kafka 主題的示例。
在創建線程池部分,他們有以下代碼
public void run(int a_numThreads) {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, new Integer(a_numThreads));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
// now launch all the threads
//
executor = Executors.newFixedThreadPool(a_numThreads);
// now create an object to consume the messages
//
int threadNumber = 0;
for (final KafkaStream stream : streams) {
executor.submit(new ConsumerTest(stream, threadNumber));
threadNumber++;
}
}
我可以向 topicCountMap 添加更多主題。 例如,
topicCountMap.put("channel1", new Integer(a_numThreads));
topicCountMap.put("channe2", new Integer(a_numThreads));
topicCountMap.put("channel3", new Integer(a_numThreads));
在上面的代碼中,在我看來,流對象只映射到主題之一
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
我不完全確定如何創建多個流對象,每個對象都映射到給定的主題,然后遍歷它們以從每個通道獲取數據並將它們提交給執行程序。
假設你有:
String topic1 = "channel1";
String topic2 = "channel2";
String topic3 = "channel3";
然后,你確實可以這樣做:
topicCountMap.put(topic1, new Integer(a_numThreads_topic1));
topicCountMap.put(topic2, new Integer(a_numThreads_topic2));
topicCountMap.put(topic3, new Integer(a_numThreads_topic3));
一旦您獲得了您的 consumerMap(執行此操作的代碼不會改變),您將能夠檢索每個主題的流:
List<KafkaStream<byte[], byte[]>> topic1_streams = consumerMap.get(topic1);
List<KafkaStream<byte[], byte[]>> topic2_streams = consumerMap.get(topic2);
List<KafkaStream<byte[], byte[]>> topic3_streams = consumerMap.get(topic3);
要從流中使用,您需要創建正確數量的執行程序:
executors_topic1 = Executors.newFixedThreadPool(a_numThreads_topic1);
executors_topic2 = Executors.newFixedThreadPool(a_numThreads_topic2);
executors_topic3 = Executors.newFixedThreadPool(a_numThreads_topic3);
最后 :
int threadNumber = 0;
for (final KafkaStream stream : topic1_streams) {
executors_topic1.submit(new ConsumerTest(streams, threadNumber));
threadNumber++;
}
for (final KafkaStream stream : topic2_streams) {
executors_topic2.submit(new ConsumerTest(stream, threadNumber));
threadNumber++;
}
for (final KafkaStream stream : topic3_streams) {
executor_topic3.submit(new ConsumerTest(stream, threadNumber));
threadNumber++;
}
當然,這只是給你的想法。 顯然,代碼可以改進。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.