Using kafka-python to consume kafka, will the local offset reset by the seek method be submitted to kafka? I am working on a solution to obtain the rpo index of the Kafka cluster in the dual-center computer room. Use kafka-python to obtain the largest timestamp of the Kafka cluster, and take the difference between the maximum timestamps of the Kafka cluster in the two computer rooms.
Use seek() to reset the offset to the maximum offset-1 of the partition, and then poll() to get the latest message, but the message can't be gotten in the loop, check the offset of the current consumer group, and find that the message is stacked Is 0
#reset offset to (max_offset-1)
for tp,offset in offsets_dict.items():
offset = offset - 1
if (offset)<0:
effective_partition = effective_partition-1
continue
consumer.seek(tp,offset)
kafkaoffset = consumer.position(tp)
if effective_partition==0:
consumer.close()
return max_timestamp
try:
Counter=0
while(True):
message = consumer.poll(max_records=1)
if not message:
continue
for partition, msgs in six.iteritems(message):
for msg in msgs:
max_timestamp = max(max_timestamp,int(msg.timestamp))
self.logger.debug(f"{max_timestamp}")
Counter = Counter +1
if Counter == effective_partition:
break
except Exception as ex:
raise ex
finally:
consumer.close()
return max_timestamp
If you commit after a seek, then that will be the new offset for the group, yes.
enable_auto_commit
defaults to True, and if you set that to False, there is a KafkaConsumer.commit(offsets)
function that you can use to manually control this behaviour
Calling .close()
also defaults to do a commit, which you should probably leave alone unless you really only wanted to seek and read a handful of messages out-of-process with a normal linear consumer process
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.