Springboot KafkaListener listening multiple topics

Question

I'm trying to build a kafka service where there are two topics and one consumer. Each topic has 10 partitions each. Currently I have a setup where the consumer listens to both the topics.

I want to make the processing priority based. I want to give more priority to events in topic 1 than that to topic 2.

So the aim is, process events in topic 2 only if there are no events in topic 1. I've been trying the Consumer.Pause option but so far had no luck as I'm a beginner in Kafka.

Appreciate a code snippet which can show what I need to achieve this..

Answer 1

Here is one solution:

@SpringBootApplication
public class So63855196Application {

    public static void main(String[] args) {
        SpringApplication.run(So63855196Application.class, args);
    }

    @Bean
    public NewTopic hiT() {
        return TopicBuilder.name("hiT").partitions(1).replicas(1).build();
    }

    @Bean
    public NewTopic loT() {
        return TopicBuilder.name("loT").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0, 10).forEach(i -> template.send("loT", "foo" + i));
            Thread.sleep(3000);
            IntStream.range(0,  2).forEach(i -> template.send("hiT", "bar" + i));
        };
    }

}

@Component
class Listener {

    private static final Logger log = LoggerFactory.getLogger(Listener.class);

    private final KafkaListenerEndpointRegistry registry;

    private final AtomicBoolean paused = new AtomicBoolean();

    Listener(KafkaListenerEndpointRegistry registry) {
        this.registry = registry;
    }

    @KafkaListener(id = "high", topics = "hiT")
    @KafkaListener(id = "low", topics = "loT",
            properties = ConsumerConfig.MAX_POLL_RECORDS_CONFIG + ":1")
    public void listen(String in, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) throws InterruptedException {
        System.out.println(in + " from " + topic);
        if (topic.equals("hiT")) {
            if (this.paused.compareAndSet(false, true)) {
                log.info("Hi-pri active; pausing low-pri");
                this.registry.getListenerContainer("low").pause();
            }
        }
        Thread.sleep(2000);
    }

    @EventListener
    public void event(ListenerContainerIdleEvent event) {
        if (event.getListenerId().startsWith("high")) {
            if (this.paused.compareAndSet(true, false)) {
                log.info("Hi-pri idle; resuming low-pri");
                this.registry.getListenerContainer("low").resume();
            }
        }
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.idle-event-interval=1000

and

foo0 from loT
foo1 from loT
bar0 from hiT
2020-09-14 09:40:28.630  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri active; pausing low-pri
bar1 from hiT
2020-09-14 09:40:37.647  INFO 39300 --- [     high-0-C-1] com.example.demo.Listener                : Hi-pri idle; resuming low-pri
foo2 from loT
foo3 from loT
foo4 from loT
foo5 from loT
foo6 from loT
foo7 from loT
foo8 from loT
foo9 from loT

This will only work if the container has concurrency=1 ; it will need a bit more sophistication when the container has multiple threads.