Messages are not delivered with kafka python

Question

I'm trying to set up a simple Kafka application with kafka-python . I've been trying to get some of the examples I found online to work but can't seem to make it. I have a kafka instance running in a docker container. I tested the shell tools and the instance is definitely working. I am able to send and receive messages. I suspect that the producer messages time out. Here are two versions of the code with basically the same behaviour:

import time
from kafka import SimpleProducer, KafkaClient
#  connect to Kafka
kafka = KafkaClient('localhost:9092')
producer = SimpleProducer(kafka)
# Assign a topic
topic = 'test'
producer.send_messages(topic, b'this is a message')

And the second version:

from kafka import KafkaProducer
from kafka.errors import KafkaError

producer = KafkaProducer(bootstrap_servers=['0.0.0.0:9092'], api_version=(0,10))
topic = "test"

producer.send(topic, b'test message')

Answer 1

Change the line: producer.send(topic, b'test message')

To: producer.send(topic, b'test message').get(timeout=30) (or any value the you see fit)

The problem is that the producer is killed before the message is sent, since this method is asynchronous. You could see it for yourself if you add:

import logging
logging.basicConfig(level=logging.INFO)

And see that the timeout is 0.

Answer 2

It depends on how you ran docker but I believe your issue is with the host name you're trying to connect to. You need to point to the host set in the ADVERTISED_HOST environment variable. For example when I run kafka-docker as docker run --hostname kafka-1 -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST='kafka-1' --env ADVERTISED_PORT=9092 spotify/kafka I produce to kafka like such

from kafka import SimpleProducer, KafkaClient

kafka = KafkaClient('kafka-1:9092')
producer = SimpleProducer(kafka)
topic = 'test'
for i in range(100):
    producer.send_messages(topic, 'hullo-' + str(i))

additionally I needed to add 127.0.0.1 kafka-1 to my /etc/hosts file. After doing this I was able to consume the messages produced by the python client with bin/kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic test --from-beginning