Kafka track offsets in all partitions

Question

Will try to explain what I am trying to achieve.

All I know is topic name and by that I must scale down to partitions. First I try

consumer.Subscribe(topics) 

And

consumer.Assignement

But if there is no delay between calls It returns empty list.

I could use consumer.Assign(..) But I dont know exact partitions, offsets yet.

Next when Iam able to go down to partition I need to get offsets low/high by time range.

For example topi "test" has 5 partitions, and I need to extract all messages info (partition, offsets) for messages being inserted from 10:00 to 10:05.

If any additional info needed, just let me know.

Thanks

Answer 1

I'm not 100% clear on what you are aiming for but some information about assignment may help here.

The assignment() method returns an empty list prior to the first time the poll method is called on a consumer when joining the group, or after a rebalance - this is because when partitions are automatically assigned the consumer only finds out the assignment as one of the steps of the poll method, prior to fetching actual records.

You can find out the actual assigned partitions either by calling poll at least once before calling assignment() - I think that is what you have discovered - else by passing a ConsumerRebalanceListener when calling subscribe(). The onPartitionsAssigned method is called during the poll - essentially a callback - with an argument that is the collection of newly assigned partitions. This enables your code to discover the current assignment before any records are fetched.

Hope this helps a bit - I have written up a blog post about this aspect of assignment but haven't yet published it - I'll add a link when I do, if it sounds like this is the issue you are facing.

Answer 2

I went for a bit different approach.

1. From IAdminClient load metada to get all available partitions for it.
2. Create TopicPartitionTimestamp with start timestamp I need to consume from.
3. Assign to TopicPartitionTimestamp and consume from it.

Also I chose to start every partition consumption on different Thread.