在同一个 Kafka 主题中监听多种类型的对象

Question

I have a use case where I would be receiving two kinds of object in the same topic. The issue I am facing is that all the objects are going to the same listener. The second listener is not being used.

My listeners are similar to below

@EnableKafka
@Service
public class consumerService {

  @KafkaListener(topic = "t", groupId = "g" containerFactory = "fooContainerFactory")
  public void consume1(Foo foo) {
     logger.info("I got : " + foo);
  }

  @KafkaListener(topic = "t", groupId = "g" containerFactory = "barContainerFactory")
  public void consume2(Bar bar) {
     logger.info("I got : " + bar);
  }

}

The container factory for Foo is below, I have a similar one for Bar.

@Configuration
public class MyConsumerConfig {


  @Value(value = "${kafka.bootstrapAddress}")
  private String bootstrapAddress;

  @Value(value = "${group.id}")
  private String groupId;

  public Map<String, Object> clickConsumerFactory() {
    /** Set the consumer properties here */
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
    return props;
  }

  @Bean
  public ConcurrentKafkaListenerContainerFactory<String, Foo> clickKafkaListenerContainerFactory() {
    /** Listener factory for the class 'Foo' */
    ConcurrentKafkaListenerContainerFactory<String, Foo> factory
            = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(new DefaultKafkaConsumerFactory<>(clickConsumerFactory()));
    factory.setMessageConverter(new StringJsonMessageConverter());
    factory.setErrorHandler(((exception, data) -> LOGGER.info("There was an error " + exception)));
    return factory;
  }
}

I tried using KafkaHandler but I don't know how to specify the individual container factory(for Foo: FooContainerFactory, for Bar: BarContainerFactory) I am stuck at this for a few hours. Can somebody tell me what should I change?

Answer 1

I could be way off here. As i am not sure how @KafkaListener works. But, off the top of my head, there are chances that there is only 1 partition and the first kakfaListener is assigned to that. so, any new listeners (similar to kafka consumers) will not get assigned any partition as there is just 1 partition to begin with. So, the new Listeners will sit idle waiting for some partition to get assigned so that they can start reading the data.

You can verify this by disabling the first KafkaListener and then running your application. The second listener would automatically kick in.

or, if your topic already has multiple partitions then, it could be happening because of the key of the message as keys can be used to send data to one partition. and your first listener is assigned to that partition.

Answer 2

You can't do that. If the same topic contains both Foo and Bar objects; there is no way for the framework to filter out the records that don't match.

And, as has been said in other comments, answers, since they are both in the same group, they will each get a subset of records.

The simplest solution is to use two different topics.

If that's not possible, you have to use a class level listener with @KafkaHandler methods, but then you need to do the conversion in the deserializer instead of the framework figuring out the type from the method parameter.

See this answer for a link to a sample that shows how to do that.