Kafka远程消费者没有收到消息

Question

I have setup 3 node Confluent/Kafka all pointing to the same zookeeper

all 3 servers have advertised.listener = public ipv4 with PLAINTEXT

server.properties:

broker.id.generation.enable=true
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://PUBLIC-IPv4:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
zookeeper.connect=10.114.16.19:2181

producer & consumer.properties files have only this value changed

bootstrap.servers=10.114.16.19:9092,10.114.16.21:9092,10.114.16.20:9092

When I run producer from remote node using node's public ip it works fine, and I can send messages & create topics, but the problem with the consumer its not getting any messages here is the code im using

Producer.py

producer = KafkaProducer(
    bootstrap_servers='PUBLIC-IP:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8'))

producer.send('slim', {'topic': 'kafka'})

Consumer.py

print('Making connection.')
consumer = KafkaConsumer(bootstrap_servers='PUBLIC-IP:9092')

print('Assigning Topic.')
consumer.assign([TopicPartition('slim', 2)])

print('Getting message.')
for message in consumer:
    print("OFFSET: " + str(message[0])+ "\t MSG: " + str(message))

when I run the Consumer py client it just stay open and doesnt get any messages Just to clarify the testing code above I found online, I didnt write it since im still learning Kafka API

Answer 1

First be sure that you can ping PUBLIC-IPPUBLIC-IP and you are not stuck in a DNS problem. Then install kafkacat if you are on a Linux platform and test if it can consume the data by the following command:

kafkacat -b <BROKER_IP> -t <TOPIC_NAME>