Externalizing Spring Kafka configuration with SpringBoot

Question

I have this kafka config class

@Configuration
public class KafkaConfiguration {
    @Value("${KAFKA_SERVERS}")
    private String kafkaServers;


    @Bean
    ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(this.consumerFactory());
        factory.setMessageConverter(new StringJsonMessageConverter());
        return factory;
    }

    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(this.consumerConfigs());
    }

    public Map<String, Object> consumerConfigs() {

        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServers);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "foo-group-id");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
        props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
        props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return props;
    }

    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServers);
        props.put(ProducerConfig.RETRIES_CONFIG, 0);
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
        props.put(ProducerConfig.LINGER_MS_CONFIG, 1);
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 33554432);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        return props;
    }

    @Bean
    public KafkaProperties.Listener listener() {
        return new KafkaProperties.Listener();
    }

    @Bean
    public KafkaTemplate<String, Map<String, Object>> kafkaTemplateHistory() {
        return new KafkaTemplate<>(new DefaultKafkaProducerFactory<>(this.producerConfigs()));
    }
}

And I want to use spring boot auto configuration so to get ride of the config class above, so I set almost all the properties in appliction.yml

but for these two properties:

ProducerConfig.LINGER_MS_CONFIG
ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG

spring.kafka.consumer doesn't propose me anything

1. So how can I set them in appliction.yml ?
2. Also how to externalize this setting line factory.setMessageConverter(new StringJsonMessageConverter()); ?

Answer 1

you can set those properties by using spring.kafka.producer.properties.*

Additional producer-specific properties used to configure the client.

spring:
   kafka:
     producer:
       properties:
         linger.ms: 10

For consumer additional properties you can use spring.kafka.consumer.properties.* with trusted package so you don't need the converter

spring:
   kafka:
     consumer:
       properties:
         sesssion.timeout.ms: 60000

And for consumer desrializer you can use this spring.kafka.consumer.value-deserializer

Deserializer class for values.

 spring:
   kafka:
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring:
          json:
            trusted:
              packages: 'learn.kafka.model'