Can you give priority to a single topic when KafkaListener listens to multiple topics?

Question

Anyone knows how to give a priority to a single kafka topic when my @KafkaListener listens to multiple topic?

Below my code example:

@KafkaListener(id = "priority", topics = { "${prio-topic}" }, concurrency = "1", autoStartup = "true")
    @KafkaListener(id = "nonPriority", topics = { "${not-prio-topic-1}",
            "${not-prio-topic-2}", "${not-prio-topic-3}",
            "${not-prio-topic-4}", concurrency = "1", autoStartup = "true")
    public synchronized void  listenManyEntryTopic(String message) {}

My problem is that i want to read from the topic prio-topic before the others nonPrio topics. And only if my prio-topic is empty I should start consuming the other topics without any specific order. Any hint/suggestion is appreciated. Thanks for the help!

Answer 1

There is no functionality in kafka to differentiate between priority vs non-priority topic messages. To solve your problem, one of the solution is to split the priority processing topic vs non-priority topics ie dedicated app1 which only consumes priority topic messages and processes them and concurrently app2 consumes non-priority messages and processes them.

Answer 2

Since you have two listener containers, you could add @Header(KafkaHeaders.RECEIVED_TOPIC) String topic as a parameter to your method.

Then, when you receive a message from the priority topic, you could stop() the other listener container (using the KafkaListenerEndpointRegistry bean), if it is running.

Configure idleEventInterval in the primary topic container and add an @EventListner method for ListenerContainerIdleEvent s (or an ApplicationListener bean).

Then, when you detect an idle primary container, you can restart the non-primary container.

EDIT

@SpringBootApplication
public class So66366140Application {

    private static final Logger LOG = LoggerFactory.getLogger(So66366140Application.class);

    public static void main(String[] args) {
        SpringApplication.run(So66366140Application.class, args);
    }

    @Bean
    public NewTopic topic1() {
        return TopicBuilder.name("so66366140-1").partitions(1).replicas(1).build();
    }

    @Bean
    public NewTopic topic2() {
        return TopicBuilder.name("so66366140-2").partitions(1).replicas(1).build();
    }

    @Autowired
    KafkaListenerEndpointRegistry registry;

    @KafkaListener(id = "so66366140-1", topics = "so66366140-1")
    @KafkaListener(id = "so66366140-2", topics = "so66366140-2", autoStartup = "false")
    public void listen(String in, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        LOG.info(in);
        if (topic.equals("so66366140-1")
                && this.registry.getListenerContainer("so66366140-2").isRunning()) {
            LOG.info("Stopping non-pri container");
            this.registry.getListenerContainer("so66366140-2").stop();
        }
    }

    @EventListener
    void events(ListenerContainerIdleEvent event) {
        LOG.info(event.toString());
        if (event.getListenerId().startsWith("so66366140-1")
                && !this.registry.getListenerContainer("so66366140-2").isRunning()) {
            LOG.info("Starting non-pri container");
            this.registry.getListenerContainer("so66366140-2").start();
        }
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            IntStream.range(0,  10).forEach(i -> {
                template.send("so66366140-1", "foo");
                template.send("so66366140-2", "bar");
                try {
                    Thread.sleep(6_000);
                }
                catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            });
        };
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.idle-event-interval=5s

You can also use pause/resume instead of stop/start.