Confluent_kafka Producer does not publish messages into topic

Question

I tried to install Kafka on my Raspberry. And test it on 'hello-kafka' topic:

~ $ /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-kafka
>Test message 1
>Test message 2
>Test message 3
>^Z
[4]+  Stopped                 /usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-kafka
$ /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello-kafka --from-beginning
Test message 1
Test message 2
Test message 3
^CProcessed a total of 3 messages

Then I tried to check that server works from another machine. Checking zookeeper:

(venv)$ telnet 192.168.1.10 2181
Trying 192.168.1.10...
Connected to 192.168.1.10.
Escape character is '^]'.
srvr
Zookeeper version: 3.6.0--b4c89dc7f6083829e18fae6e446907ae0b1f22d7, built on 02/25/2020 14:38 GMT
Latency min/avg/max: 0/0.8736/59
Received: 10146
Sent: 10149
Connections: 2
Outstanding: 0
Zxid: 0x96
Mode: standalone
Node count: 139
Connection closed by foreign host.

And Kafka:

(venv) $ telnet 192.168.1.10 9092
Trying 192.168.1.10...
Connected to 192.168.1.10.
Escape character is '^]'.
tets
Connection closed by foreign host.

Then I wrote a Python script:

# -*- coding: utf-8 -*-

from confluent_kafka import Producer


def callback(err, msg):
    if err is not None:
        print(f'Failed to deliver message: {str(msg)}: {str(err)}')
    else:
        print(f'Message produced: {str(msg)}')


config = {
            'bootstrap.servers': '192.168.1.10:9092'
        }

producer = Producer(config)
producer.produce('hello-kafka', value=b"Hello from Python", callback=callback)
producer.poll(5)

There is script output (no any prints):

(venv) $ python kafka-producer.py 
(venv) $ python kafka-producer.py 
(venv) $ 

And no new messages in Kafka:

$ /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello-kafka --from-beginning
Test message 1
Test message 2
Test message 3
^CProcessed a total of 3 messages
$ ^C

Somebody can tell me what I am doing wrong?

Answer 1

The correct fix is to update your broker configuration in server.properties to set the advertised listener correctly. If your client cannot resolve raspberrypi then change the advertised listener to something that your client can reach, ie the IP address:

advertised.listeners=PLAINTEXT://192.168.1.10:9092

Changing the /etc/hosts file on your client is a workaround that for a test project with a Raspberry Pi is fine, but as a general best practice should be discouraged (because the client will break as soon as it's moved to another machine which doesn't have the /etc/hosts hack)

Answer 2

I turned on log and saw next message:

WARNING:kafka.conn:DNS lookup failed for raspberrypi:9092, exception was [Errno 8] nodename nor servname provided, or not known. Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and resolvable?
ERROR:kafka.conn:DNS lookup failed for raspberrypi:9092 (AddressFamily.AF_UNSPEC)

Then I added to /etc/hosts on client machine next string:

192.168.1.10 raspberrypi

And it completely fix this situation.