Confluent Python API for Kafka

Question

I'm getting an error with basic usage of the official Confluent Kafka Python API:

I subscribe:

kafka_consumer.subscribe(topics=["my-avro-topic"], on_assign=on_assign_callback, on_revoke=on_revoke_callback)

Using callback:

def on_assign_callback(consumer, topic_partitions):
for topic_partition in topic_partitions:
    print("without position. topic={}. partition={}. offset={}. error={}".format(topic_partition.topic, topic_partition.partition,
                                   topic_partition.offset, topic_partition.error))

topic_partitions_with_offsets = consumer.position(topic_partitions)
print("assigned to {}->{} partitions".format(len(topic_partitions), len(topic_partitions_with_offsets)))

for topic_partition in topic_partitions_with_offsets:
    print("with position. topic={}. partition={}. offset={}. error={}".format(topic_partition.topic, topic_partition.partition,
                                   topic_partition.offset, topic_partition.error))

which produces the console output:

without position. topic=my-avro-topic. partition=0. offset=-1001. error=None
assigned to 1->1 partitions
with position. topic=my-avro-topic. partition=0. offset=-1001. error=KafkaError{code=_UNKNOWN_PARTITION,val=-190,str="(null)"}

Can someone explain this? Why would I get a callback notification on an unknown partition? Similar code works perfectly using the Java API.

Answer 1

This is a bug in the underlying C library librdkafka. See the upstream issue .

If you want to start consuming from the stored offsets you don't actually need to call position() to retrieve them, the client will do so automatically if you don't change the default offset of -1001.