在 Python 上的服务器之间发送数据 Kafka Apache

Question

producer = KafkaProducer(bootstrap_servers='kf-p1l-node3:9092,xxxxx,xxxxx',
                                 value_serializer=lambda x: dumps(x).encode('utf-8'))  # utf-8

consumer = KafkaConsumer(  bootstrap_servers='rdwh-node1:49092,xxxxx,xxxxx',
                                 # bootstrap_servers='kf-p1l-node3:9092,xxxxx,xxxxx',
                                 auto_offset_reset=param["AUTO_OFFSET_RESET"],
                                 consumer_timeout_ms=param["CONSUMER_TIMEOUT_MS"],
                                 enable_auto_commit=False,
                                 auto_commit_interval_ms=60000,
                                 group_id=param["GROUP_ID"],
                                 client_id=param["CLIENT_ID"]
                                 )
consumer.subscribe([param["TOPIC_IN"]])

This code work if, KafkaProducer and KafkaConsumer's bootstrap_server are the same. But if change KafkaConsumer to another server it doesn't work

Answer 1

Bootstrap servers must contain all servers for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping. You can check the documentation here: http://kafka.apache.org/090/documentation.html

consumer = KafkaConsumer('my-topic',
                         group_id='my-group',
                         bootstrap_servers=['node1:port1', 'node1:port2', 'node2:port3'])