I am working on Kafka with Spark streaming in Scala. There are multiple topics from which I need to fetch messages. For that, I need to input consumer group of each topic.
I have tried following command:
bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group group_name
For this, I need to mention consumer group and check whether my topic falls under this group. But I don't know the consumer group value for some topics.
So, is there any command or programmatic way of finding consumer group for a given topic?
Any help on this would be greatly appreciated. Thank you
You can read data from various topics by specifying the topics. You can simply create a Map of topics. Refer to the below link
https://www.tutorialspoint.com/apache_kafka/apache_kafka_consumer_group_example.htm
There is no Command/API provided by kafka to list consumer groups consuming from a specified topic. Instead you can use a tool developed by yahoo to manage kafka clusters : Yahoo Kafka Manager . After installing this software and pointing yahoo kafka manager to your kafka cluster, you can view consumer groups consuming from a specified topic.
for i in `kafka-consumer-groups --list --bootstrap-server localhost:9092`; do echo $i; (kafka-consumer-groups --describe --bootstrap-server localhost:9092 --group $i 2>&1| grep mytopic); done
这需要一些时间,因为它会检查所有主题,但会告诉哪些消费者群体使用该主题。
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.