Python Confluent-kafka DeserializingConsumer is not reading the messages from Kafka topic, even though the messages are present

Question

I have a topic with avro schema, I am producing the messages via Python code and it works completely fine. When I consume the messages from CLI, I can consume them successfully without errors.

When I am trying to consume via Python code, it prints 'None', basically it tries to read but gets none, I tried to print the offset and it throws '-1001'.

The method aims to read all the latest messages, create a list with those messages and returns the list.

Note:- I have also tried usinng 'enable.auto.commit' = True, but it didn't work so removed it from my config.

Library in requirement.txt = confluent-kafka[avro]>=1.4.2

   conf = {
    'bootstrap.servers': 'dummyvalue',
    'security.protocol': 'dummyvalue',
    'sasl.mechanism': 'PLAIN',
    'sasl.username': 'dummyvalue',
    'sasl.password': 'dummyvalue',
    'session.timeout.ms': 45000,
    'schema.registry.url': 'dummyvalue',
    'basic.auth.credentials.source': 'dummyvalue',
    'basic.auth.user.info': 'dummyvalue',
    'use.latest.version': True
} 



 schema_registry_conf = {
    'url': conf['schema.registry.url'],
    'basic.auth.user.info': conf['basic.auth.user.info']
 }



    def _set_consumer_config(self, conf, avro_deserializer):
        consumer_conf = self._popSchemaRegistryParamsFromConfig(conf) 
        #above method will remove unnecessary configs from main conf dictionary so consumer_conf has only relevant properties
        consumer_conf['value.deserializer'] = avro_deserializer
        consumer_conf['group.id'] = "python_example"
        consumer_conf['auto.offset.reset'] = 'latest'    
        return consumer_conf

    
    def get_list_of_unconsumed_msgs(self, topic):

        text_file = open('avro schema file path')
        avro_schema = text_file.read()
        schema_registry_client = SchemaRegistryClient(schema_registry_conf)
        avro_deserializer = AvroDeserializer(schema_registry_client,avro_schema)
        consumer = DeserializingConsumer(self._set_consumer_config(conf, avro_deserializer))
        consumer.subscribe([topic])
        messages = []
        polling_count = 5
        while polling_count >= 1:
            try:
                print(consumer.position([TopicPartition(topic, 0)]))
                print(f"Consumer Committed {consumer.committed([TopicPartition(topic, 0)])}")
                print(f"Consumer Assignment {consumer.assignment()}")
                msg = consumer.poll(3.0)
                if msg is None:
                    polling_count = polling_count - 1
                    continue
                elif msg.error():
                    print('error: {}'.format(msg.error()))
                else:
                    messages.append([msg.value()])
            except SerializerError as e:
                # Report malformed record, discard results, continue polling
                print("Message deserialization failed {}".format(e))
        consumer.close()
        return messages

    def main():
        msg = {}
        topic_name = "aa_automation_test"
        msg = obj.get_list_of_unconsumed_msgs(topic)
        print(f"Received Message as :- {msg}")

Output of print statement:

[Prints an empty list, for debugging I have printed offset and it throws -1001]
[TopicPartition{topic=aa_automation_test,partition=0,offset=-1001,error=None}]
Consumer Committed [TopicPartition{topic=aa_automation_test,partition=0,offset=-1001,error=None}]
Consumer Assignment [] 
Received Message as :- []

Answer 1

You're seeing default values.

There won't be any data until you actually poll and connect to the brokers.

Kafka keeps track of uncommitted offsets on its own. You don't need to implement that logic in your app.

If you print an empty list at the end, from what you've shown, that means you have reached the end of the topic.

To pull 5 at a time, see consume(num_messages)

To check the (end) offsets for partitions, get_watermark_offsets , which you can subtract off from consumer.committed() to see lag.