Multiple consumers with same group id in Python

Question

Does anybody know how to run multiple consumers with same group id in Python? I've tried following

a = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
b = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
a.subscribe([topic_to_read])
b.subscribe([topic_to_read])
c.subscribe([topic_to_read]) 
running = True
while running:
    msg1 = a.poll(timeout=timeout)
    msg2 = b.poll(timeout=timeout)
    msg3 = c.poll(timeout=timeout)

But this is not working. So I've tried using multiprocessing lib but I am not able to make it work.

Answer 1

A group ID is a unique ID to each consumer. If you are subscribing to a same topic with multiple consumers, you must have different group ID's or else only one of those consumers will get the messages.

Answer 2

Check how many partitions you have for that topic. The number of consumers within a group id should not exceed the number of partitions of the topic the group is consuming from. Else the extra consumers will remain idle. ALSO CHECK IF YOU ARE ASSIGNING DIFFERENT CLIENID/CONSUMERID FOR EACH OF THE CONSUMERS.