Reading oldest available message in Kafka using KafkaConsumer instance of kafka-python kafka client

Question

I try to read messages in kafka consumer using the following command:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Here we can read old messages of about 4 days as we have set retention time in kafka server configuration file as 7 days. But while we are trying to read messages using KafkaConsumer of kaka-python client library like following:

cons = KafkaConsumer("localhost:9092", "test","smallest")
cons.fetch_messages()

we are getting messages of today's only with some offset. I don't have any idea how to get oldest message available in Kafka like we got in kafka consumer shell script above. Please help.

Answer 1

The docs show the config passed in via namedtuples .

consumer = KafkaConsumer('topic1', 'topic2',
                         bootstrap_servers=['localhost:9092'],
                         group_id='my_consumer_group',
                         auto_commit_enable=True,
                         auto_commit_interval_ms=30 * 1000,
                         auto_offset_reset='smallest')