spring boot kafka中基于consumer record key的多个consumer

Question

I want to list kafka topic based on the consumer record key. Right now I have consumer and based on the condition I am processing the record

@KafkaListener(topics = "topic",
        containerFactory = "kafkaListenerStringContainerFactory")
public void listenasString(ConsumerRecord<String, String> cr) {
  if (cr.key().equals("A")) {
    //do some processing
  } else if (cr.key().equals("B")) {
    //do some processing
  }

}

Is it possible to have multiple consumers to listen to same topic based on the key?

Answer 1

I'm not sure what you want is possible

have multiple consumers to listen to same topic

Yes, this is possible

based on the key

This is not, because a key is not a queryable. You would need to compute the partition of that key , then you would need to assign the consumer to that partition .

Outside of Spring, you can initialize DefaultPartitioner and use consumer.assign to make this work

From there I would suggest using one consumer thread per partition, and your secondary if statements would need to move unless there's multiple keys in the same partitions (which is possible)

Answer 2

将 kafka 侦听器并发设置为大于 1

spring.kafka.listener.concurrency=3 

Answer 3

You can use streams to create two topics based on keys and run a consumer for each of these topics. But if you want that to be in the same topic then @criket_007 answer is correct, this is not possible.

Check Kafka Stream Branch https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#stateless-transformations