Kafka consumer offset auto-reset and other parameters

Question

I have a KafkaConsumer which needs to subscribe to two topics topicA and topicB . However I need some parameters be different. Eg if I need auto.offset.reset for topicA be earilest while for topicB it should be latest . I see no easy way to do that. One option is to run two consumers, but in this case I need two polling threads for them and hence should handle multi-threading. Is there any simpler way?

Answer 1

Creating two (or more) threads is correct.

Consumers are not thread-safe, and should be isolated and separated from other processes anyway.

You could use higher-level Kafka libraries (eg Vert.x / Spring) for simplifying this.

Answer 2

If you need to use one consumer, then I think you could set auto.reset.offset to latest and then move offsets of topicA manually (if needed). To do so, between subscription and poll loop you could:

1. Get partitions assigned to the consumer (method assignment )
2. Check commited offsets of topic topicA by filtering partitions of topicA from the previous point (method committed ). If the result is null , then call seekToBeginning method.