跨Kafka分区对消息进行排序并将其放入另一个Kafka主题中

Question

I have a Kafka topic with X partitions. Each message has a timestamp, ts . Can someone suggest me some way of sorting all the messages (based on ts ) across all partitions and putting it in a new topic with Y partitions ( Y < X where Y can be 1 also)?

During this operation, no new data will be added to the original Kafka topic. I am trying to avoid buffering all data to a temporary data store to sort. So basically I am looking for a X-Way merge on streaming data.

Can someone let me know if this is possible to do efficiently in java using Kafka Streams API?

Answer 1

This is my best suggestion based on my last experience, since you do not want to buffer all in one place, you can take a time interval say 30 mins, so you pull all data from partitions until you are getting data within that time frame say 9.00 am to 9.30 AM and sort this and put to the target, next you start pulling next data which would be from 9:30 AM on-wards. Although after 9:30 data there is a possibility u may get 9:27 data due to delays in your data and hence after processing this batch you have possibility of one data point of 9:29 and another of 9:27 however you will find that all of the data between 9:10 to 9:20 is sorted. Now the higher the time frame you can take more is the accuracy. If you need 100% sorting u may have to iterate on this target data again with different data frame to sort further.