Kafka Consumer not fetching all messages

Question

I am trying to launch dynamic consumer whenever new topic is created in Kafka but dynamically launched consumer is missing starting/first message always but consuming the message from there on. I am using kafka-python module and am using updated KafkaConsumer and KafkaProducer.

Code for Producer is

producer = KafkaProducer(bootstrap_servers='localhost:9092')
record_metadata = producer.send(topic, data)

and code for consumer is

consumer = KafkaConsumer(topic,group_id="abc",bootstrap_servers='localhost:9092',auto_offset_reset='earliest')

Please suggest something to over come this problem or any configuration i have to include in my producer and consumer instances.

Answer 1

Can you set auto_offset_reset to earliest.

When a new consumer stream is created, it starts from latest offset (which is default value for auto_offset_reset) and you will miss messages which were sent while consumer wasn't started.

You can read about it in kafka python doc . Relevant portion is below

auto_offset_reset (str) – A policy for resetting offsets on OffsetOutOfRange errors: 'earliest' will move to the oldest available message, 'latest' will move to the most recent. Any ofther value will raise the exception. Default: 'latest'.