kafka consumer java with multiple topics

Question

we have one consumer group and three topics, all three topics are of different schema . created one consumer with a for loop passing each topic at a time and polling it processing and committing manually. Method used is consumer created common and in for loop I am subscribing one topic at a time and processing data. I am seeing a random lag of consumer , although the topic has data my consumer fetches no records from topic and fetches sometimes. When I work out with a single topic instead of looping through three topics it is working but unable to reproduce. need help to debug the issue and reproduce the same,

Answer 1

Rather than looping three topics in a single method, you could create a skeleton thread like so that consumes from any topic. See examples here

I can't say if this will "fix" the problem, but trying to consume from topics with different schemas in one application is usually not a scalable pattern, but it's not really clear what you're trying to do.

class ConsumerThread extends Thread {

    KafkaConsumer consumer;
    AtomicBoolean stopped = new AtomicBoolean();

    ConsumerThread(Properties props, String subscribePattern) {
        this.consumer = new KafkaConsumer...
        this.consumer.subscribe(subscribePattern);
    } 

    @Override
    public void run() {
        while (!this.stopped.get()) {
            ... records = this.consumer.poll(100);
            for ( ... each record ... ) {
               // Process record
            } 
        }
    }

    public void stop() {
        this.stopped.set(true);
    }
}

Not meant to be production-grade

Then run three consumers independently.

new ConsumerThread("t1").start();
new ConsumerThread("t2").start();
new ConsumerThread("t3").start();

Note : KafkaConsumer is not thread-safe.