Handling a producer and consumer using Kafka-python

Question

First I would like to say that I am a newbie to Kafka and also stackoverflow, So I am sorry if I am not asking this in the right way. I am trying to implement the producer-consumer using kafka-python. But its not working properly

I have the zookeeper installed and its up and running. I have up the kafka-server also. But when I am running the consumer and producer through pycharm, the messages are not getting received by the receiver.The consumer keeps on running but the producer stops.

consumer.py

from kafka import KafkaConsumer

consumer = KafkaConsumer('test', group_id='test-consumer-group', 
bootstrap_servers=['my_ip:9092'], api_version=(0, 10, 1),
                     auto_offset_reset='earliest')
print("Consuming messages")
for msg in consumer:
    print(msg)

producer.py

from kafka import KafkaProducer

print('above producer')
producer = KafkaProducer(bootstrap_servers=['my_ip:9092'], api_version=(0, 10, 1),
                         compression_type=None
                         )
print('after producer')
for _ in range(100):
    producer.send('test', b'HELLO NITHIN chandran')

print('after sending messages')

In the place of my_ip, I have provided with my system ip address from ipconfig.

consumer.py Output -

Consuming messages

The consumer.py doesnt stop running

producer.py Output -

above producer
after producer
after sending messages

Process finished with exit code 0

The producer.py stops running and the process is finished as shown in the output.

Please help me in resolving this issue. All help are appreciated

Answer 1

Your code is ok, the problem is about your broker configuration. Please set it to initial configuration, just change the log.dirs to the path that you want to store Kafka data. After changing the config file follow these steps:

• Stop zookeeper and kafka
• Clear both kafka and zookeeper data dir
• Run zookeeper and kafa
• Start consumer and producer