For some data processing, we need to reprocess all the messages between 2 timestamps say between 1st Jan to 15th Jan.
to control upper bound we are planning to create a new topic that will have these messages so that once this task is complete, we can delete the topic too. The new topic will have data from a particular offsets of source topic
partition 1 - from offset 100
partition 2 - from offset 2400... and so on
What is the most suitable solution for this? approx 10lacs messages fall in this.
.assign
for the partitions you want to copy.seek
for each starting offset of those partitions. You can use offsetsForTimes
method to get them for a specific timestamp; then you can pass those on to the seek method.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.