How to see (new-) consumers connected to a specific Kafka topic

Question

I have a list of new-consumers (python consumers). I can retrieve the groups with this command:

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list

I can obtain for each one to which topic they are connected to

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092  --describe --group TheFoundGroupId

1. How do can I obtain all groups (preferably all consumers even when not in a group) that are connected to a topic?
2. Is there a way to access this from python other than running this as a shell command?

Answer 1

Thanks for asking this question.

all the consumer configuration like consumer group ids, which consumer is subscribed to which topic is stored in zookeeper.

Run below command to get connected to zookeeper

./bin/zookeeper-shell localhost:2181

and then run

ls /consumers

you will get all the consumer groups which are present. if you don't provide consumer group also. Kafka will allocate random consumer group. for console consumers it will allocate console-consumer-XXXXX id

you can get all consumer groups from below python snippet

install zookeeper python client

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()


# get all consumer groups
consumer_groups = zk.get_children("/consumers")
print("There are %s consumer group(s) with names %s" % (len(consumer_groups), consumer_groups))

# get all consumers in group
for consumer_group in consumer_groups:
    consumers = zk.get_children("/consumers/"+consumer_group)
    print("There are %s consumers in %s consumer group. consumer are : %s" % (len(consumers), consumer_group, consumers))

To get consumers or consumer groups which are connected to a topic.

get /consumers/consumergroup_id/ids/consumer_id/

will give you output like

{"version":1,"subscription":{"test":1},"pattern":"white_list","timestamp":"1514218381246"}

under subscription object all the topics subscribed by consumer. implement logic according to your use case

Thanks

Answer 2

It's not the best solution but since nobody seems to have an answer, here is how I solved it in the end (assign a group to the consumer and replace ___YOURGROUP____):

    import subprocess
    import os
    if "KAFKA_HOME" in os.environ:
        kafkapath = os.environ["KAFKA_HOME"]
    else:
        kafkapath = oms_cfg.kafka_home
        # error("Please set up $KAFKA_HOME environment variable")
        # exit(-1)

    instances = []
    # cmd = kafkapath + '/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server {} --list'.format(oms_cfg.bootstrap_servers)
    # result = subprocess.run(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    igr = ____YOURGROUP_____ # or run over all groups from the commented out command
    print("Checking topics of consumer group {}".format(igr))
    topic_cmd = kafkapath + '/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server ' + oms_cfg.bootstrap_servers + ' --describe --group {gr}'
    result = subprocess.run(topic_cmd.format(gr=igr).split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    table = result.stdout.split(b'\n')
    # You could add a loop over topic here
    for iline in table[1:]:
        iline = iline.split()
        if not len(iline):
            continue
        topic = iline[0]
        # we could check here for the topic. multiple consumers in same group -> only one will connect to each topic
        # if topic != oms_cfg.topic_in:
        #     continue
        client = iline[-1]
        instances.append(tuple([client, topic]))
        #    print("Client {} Topic {} is fine".format(client, topic))
    if len(instances):
        error("Cannot start. There are currently {} instances running. Client/topic {}".format(len(instances),
                                                                                                  instances))
        exit(-1)