简体   繁体   中英

Using kafka-python to consume kafka, will the local offset reset by the seek() be submitted to kafka?

Using kafka-python to consume kafka, will the local offset reset by the seek method be submitted to kafka? I am working on a solution to obtain the rpo index of the Kafka cluster in the dual-center computer room. Use kafka-python to obtain the largest timestamp of the Kafka cluster, and take the difference between the maximum timestamps of the Kafka cluster in the two computer rooms.

Use seek() to reset the offset to the maximum offset-1 of the partition, and then poll() to get the latest message, but the message can't be gotten in the loop, check the offset of the current consumer group, and find that the message is stacked Is 0

#reset offset to (max_offset-1)
for tp,offset in offsets_dict.items():
            offset = offset - 1
            if (offset)<0:
                effective_partition = effective_partition-1
                continue
            consumer.seek(tp,offset)
            kafkaoffset = consumer.position(tp)
           
        if effective_partition==0:
            consumer.close()
            return max_timestamp

        try:
            Counter=0
            while(True):
                message = consumer.poll(max_records=1)
                if not message:
                    continue
                for partition, msgs in six.iteritems(message):
                    for msg in msgs:
                        max_timestamp = max(max_timestamp,int(msg.timestamp))
                        self.logger.debug(f"{max_timestamp}")
                Counter = Counter +1
                if Counter == effective_partition:
                    break
        except Exception as ex:
            raise ex
        finally:
            consumer.close()
            return max_timestamp

If you commit after a seek, then that will be the new offset for the group, yes.

enable_auto_commit defaults to True, and if you set that to False, there is a KafkaConsumer.commit(offsets) function that you can use to manually control this behaviour

Calling .close() also defaults to do a commit, which you should probably leave alone unless you really only wanted to seek and read a handful of messages out-of-process with a normal linear consumer process

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM