如何查看（新）与特定Kafka主题相关的消费者

Question

我有一个新消费者列表（python消费者）。 我可以使用此命令检索组：

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list

我可以为每个人找到他们所连接的主题

bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092  --describe --group TheFoundGroupId

1. 如何获得与主题相关的所有群组（最好是所有消费者，即使不在群组中）？
2. 除了将它作为shell命令运行之外，有没有办法从python访问它？

Answer 1

谢谢你提出这个问题。

所有消费者配置，如消费者组ID，消费者订阅哪个主题存储在zookeeper中。

运行以下命令以连接到zookeeper

./bin/zookeeper-shell localhost：2181

然后跑

ls /消费者

您将获得所有在场的消费者群体。 如果你不提供消费者群体。 卡夫卡将分配随机消费者群体。 对于控制台用户，它将分配console-consumer-XXXXX id

你可以从python片段下面获得所有的消费者群体

安装zookeeper python客户端

from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()


# get all consumer groups
consumer_groups = zk.get_children("/consumers")
print("There are %s consumer group(s) with names %s" % (len(consumer_groups), consumer_groups))

# get all consumers in group
for consumer_group in consumer_groups:
    consumers = zk.get_children("/consumers/"+consumer_group)
    print("There are %s consumers in %s consumer group. consumer are : %s" % (len(consumers), consumer_group, consumers))

获取与主题相关联的消费者或消费者群组。

get / consumers / consumergroup_id / ids / consumer_id /

会给你输出像

{"version":1,"subscription":{"test":1},"pattern":"white_list","timestamp":"1514218381246"}

在订阅对象下，消费者订阅的所有主题。 根据您的用例实现逻辑

谢谢

Answer 2

这不是最好的解决方案，但由于似乎没有人有答案，所以我最终解决了这个问题（将一个小组分配给消费者并替换___YOURGROUP____）：

    import subprocess
    import os
    if "KAFKA_HOME" in os.environ:
        kafkapath = os.environ["KAFKA_HOME"]
    else:
        kafkapath = oms_cfg.kafka_home
        # error("Please set up $KAFKA_HOME environment variable")
        # exit(-1)

    instances = []
    # cmd = kafkapath + '/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server {} --list'.format(oms_cfg.bootstrap_servers)
    # result = subprocess.run(cmd.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    igr = ____YOURGROUP_____ # or run over all groups from the commented out command
    print("Checking topics of consumer group {}".format(igr))
    topic_cmd = kafkapath + '/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server ' + oms_cfg.bootstrap_servers + ' --describe --group {gr}'
    result = subprocess.run(topic_cmd.format(gr=igr).split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    table = result.stdout.split(b'\n')
    # You could add a loop over topic here
    for iline in table[1:]:
        iline = iline.split()
        if not len(iline):
            continue
        topic = iline[0]
        # we could check here for the topic. multiple consumers in same group -> only one will connect to each topic
        # if topic != oms_cfg.topic_in:
        #     continue
        client = iline[-1]
        instances.append(tuple([client, topic]))
        #    print("Client {} Topic {} is fine".format(client, topic))
    if len(instances):
        error("Cannot start. There are currently {} instances running. Client/topic {}".format(len(instances),
                                                                                                  instances))
        exit(-1)