Send data between servers Kafka Apache on Python

Question

producer = KafkaProducer(bootstrap_servers='kf-p1l-node3:9092,xxxxx,xxxxx',
                                 value_serializer=lambda x: dumps(x).encode('utf-8'))  # utf-8

consumer = KafkaConsumer(  bootstrap_servers='rdwh-node1:49092,xxxxx,xxxxx',
                                 # bootstrap_servers='kf-p1l-node3:9092,xxxxx,xxxxx',
                                 auto_offset_reset=param["AUTO_OFFSET_RESET"],
                                 consumer_timeout_ms=param["CONSUMER_TIMEOUT_MS"],
                                 enable_auto_commit=False,
                                 auto_commit_interval_ms=60000,
                                 group_id=param["GROUP_ID"],
                                 client_id=param["CLIENT_ID"]
                                 )
consumer.subscribe([param["TOPIC_IN"]])

This code work if, KafkaProducer and KafkaConsumer's bootstrap_server are the same. But if change KafkaConsumer to another server it doesn't work

Answer 1

Bootstrap servers must contain all servers for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping. You can check the documentation here: http://kafka.apache.org/090/documentation.html

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['node1:port1', 'node1:port2', 'node2:port3'])