Kafka Java使用者

Question

My Kafka consumer has 2 threads and the number of partitions is let's say 10 so overall 5 partitions per consumer thread. I am saving the time at which a particular record needs to be processed. Now if record1 on partition1 needs to be picked 10 hours from now Thread should move to next partition to see if next partition can be picked.

example :

P1 - 8
P2 - 7
P3 - 6
P4 - 5
P5 - 4

Now data on partition P1 needs to be picked at 8 hours and the current time is 6 hours if I make my thread to wait for 8 hours I'll wait for 1 hour although I could process P3, P4, and P5.

Please let me know how should I proceed.

Answer 1

I would go easier way: have consumer per partition. There is KafkaConsumer.assign() API to really select and assign particular partition to the consumer. This way each of them will deal with its own isolated logic based on the mentioned schedule.