How to create a Kafka consumer library to consume multiple topics

Question

I have 'N' number of Kafka topics, where I need to consume messages from different topics and based on the business logic. I need to process/filter them and send it to service bus.I don't want to create a N number of kafka configurations to consume messages.I just want to make it as a library, where I can simply externalize the properties to pick and configure the consumers.So that I can just put the business logic inside my application. Did anyone have already done this kind of implementations before.Please let me know the best practices for this approach.

Edited: Here is my KafkaConsumerConfig looks like:

KafkaConsumerConfig.java

@Configuration
public class KafkaConsumerConfig {

    @Value("${spring.kafka.consumer.bootstrap-servers}")
    private String bootstrapServers;

    @Value("${spring.kafka.consumer.key-deserializer}")
    private String keyDeserializer;

    @Value("${spring.kafka.consumer.value-deserializer}")
    private String valueDeserializer;

    @Value("${spring.kafka.consumer.group-id}")
    private String groupIdConfig;

    @Value("${spring.kafka.consumer.auto-offset-reset}")
    private String autoOffsetResetConfig;

    @Value("${spring.kafka.consumer.client-id}")
    private String clientIdConfig;

    @Bean
    public KafkaReceiver receiver() {
        return new KafkaReceiver();
    }

    @Bean
    public ConsumerFactory<?, ?> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> properties = new HashMap<>();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, keyDeserializer);
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, valueDeserializer);
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, groupIdConfig);
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, autoOffsetResetConfig);
        return properties;
    }

}

Answer 1

You can subscribe to any number of topics with a single consumer via

KafkaConsumer#subscribe(Collection<String> topics)

Answer 2

If you are consuming different payloads from a topic then you can simply use @KafkaListener on a Class and can have their own path for each type

Example:

@KafkaListener(id = "multi", topics = "myTopic1,mytopic2")
public class MultiListenerBean {

    @KafkaHandler
    public void listen(String foo) {

    }

    @KafkaHandler
    public void listen(Integer bar) {

    }

   @KafkaHandler(isDefault = true`)
   public void listenDefault(Object object) {

   }

 }

Starting with version 2.1.3, you can designate a @KafkaHandler method as the default method that is invoked if there is no match on other methods. At most, one method can be so designated. When using @KafkaHandler methods, the payload must have already been converted to the domain object (so the match can be performed). Use a custom deserializer, the JsonDeserializer, or the JsonMessageConverter with its TypePrecedence set to TYPE_ID. See Serialization, Deserialization, and Message Conversion for more information.

Note: This approach will not work for batch listeners