How to consume data from multiple brokers of kafka in python?

Question

I created the topic with this command

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic replica

Then i run this command

bin/kafka-console-producer.sh --broker-list localhost:9093 localhost:9094 --topic replica

and I am able to get message in consumer with this command

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic replica

But I am unable to do same in python

In python I set this on producer side

producer = KafkaProducer(bootstrap_servers=['localhost:9093', 'localhost:9094'],
                         value_serializer=lambda x:
                         dumps(x).encode('utf-8'))

producer.send('replica', value=data_obj)

and on consumer side I set this

from pprint import pprint
def subscriber(topic):

    consumer = KafkaConsumer(
    topic,
     bootstrap_servers=['localhost:9093', 'localhost:9094'],
     auto_offset_reset='earliest',
     enable_auto_commit=True,
     group_id='my-group',
     value_deserializer=lambda x: loads(x.decode('utf-8')))

    for msg in consumer:

        pprint(msg)


if __name__ == '__main__':


   subscriber('replica')

What is the problem? Why I am not able to consume data?

Answer 1

还要添加 localhost:9092.That 是您的引导服务器并尝试像这样替换 --replication-factor 3

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic replica

Answer 2

As far as i'm concerned Kafka-Python module does not support multiple brokers
If you build your kafka cluster with just a broker you can see it's gonna work but instead you can use Confluent-Kafka module here is doc: https://docs.confluent.io/platform/current/clients/confluent-kafka-python/