简体   繁体   中英

Consuming multiple kafka topics in the same consumer class

I have a spring boot project and I am spring-kafka to connect to the underlying kafka event hub.

I have to listen to 2 different topics in the same consumer class. I have two approaches to do so.

One is to have two kafka listeners like this:

 @KafkaListener(topics = "topic1")
public void consumeTopic1(String message) throws Exception {  
   //do something
  }

 @KafkaListener(topics = "topic2")
public void consumeTopic2(String message) throws Exception {  
   //do something
  }

Another approach is to have 2 topics in the same kafkaListener like this

 @KafkaListener(topics = {"topic1", "topic2"})
public void consumeTopics(String message) throws Exception {  
   //do something
  }

Since I am new to kafka, I am not sure what is the difference between the two approaches. Which one is performant and resource effective.

One thing I am wondering about is that would it listen to both the topics on a single thread in both approaches or it would spawn a thread each to listen to these topics.

Using approach 1, I have had troubles with the consumer where I see some delays in the consumption of the topics.

Please suggest me based on your experience as I am pretty new to kafka

===============EDIT======== Kafka properties are in application.yml as follows:

 kafka:
    properties:
      topics:
        topic1: topic1
        topic2: topic2
    bootstrap-servers: server1,server2
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      retries: 4
    consumer:
      group-id: mygroupid
      auto-offset-reset: latest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
  • The purpose of creating multiple topics is to spread the data, and
    subsequently, the processing of data, across multiple
    cores/processes/threads.

  • Even then, you can start with a single consumer for multiple topics until a delay is observed in processing events in queue. Then you can separate them out.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM