Spring Boot with Kafka : How to write unit Integration tests excluding kafka configuration

Question

I am using spring-kafka dependency in my SpringBoot application to use Kafka.

Everything works fine until my Kafka instance is up and running, but the problem is my unit&integration tests, they run fine in my local but not in my deployment pipeline ( which is obvious since Application is trying to connect with Kafka instance [in my build pipeline] while running tests and unable to find any ), so end up getting following error :

[Consumer clientId=consumer-1, groupId=biz-web-group-test] Connection to node -1 
could not be established. Broker may not be available.

This is only happening until I have a method annotated with @KafkaListener annotation

 @KafkaListener(topics = "${biz-web.kafka.message.topic.name}", groupId = "${biz-web.kafka.message.group.id}")
 public void listenToKafkaMessages(ConsumerRecord consumerRecord) {
        // Some Logic
 }

As soon as I comment // this annotation, test cases work fine.

Is there a way I can exclude this kafka-related configuration or this annotation while running the unit/integration tests.

Answer 1

As pointed out, spring-test-kafka is the library to be used in the unit/integration tests.

EmbeddedKafkaBroker : An embedded Kafka Broker(s) and Zookeeper manager. This class is intended to be used in the unit tests.

EmbeddedKafkaTest.java

@RunWith(SpringRunner.class)
@SpringBootTest
@EmbeddedKafka(topics= {"test"},count=1,partitions=1)
public class EmbeddedKafkaTest {

    private static final Logger logger = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());

    @Autowired
    private EmbeddedKafkaBroker embeddedKafka;

    private KafkaTemplate<Integer, String> kafkaTemplate;


    @Test
    public void testConsumer() {

        Map<String, Object> producerProps=KafkaTestUtils.producerProps(this.embeddedKafka);
        ProducerFactory<Integer, String> producerFactory = new DefaultKafkaProducerFactory<>(producerProps);
        this.kafkaTemplate = new KafkaTemplate<>(producerFactory);
        logger.info("embedded kafka ",embeddedKafka);
        kafkaTemplate.send("test", "hello");
         Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("demo-group", "true", this.embeddedKafka);
        consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        ConsumerFactory<Integer, String> cf = new DefaultKafkaConsumerFactory<>(consumerProps);
        Consumer<Integer, String> consumer = cf.createConsumer();
        this.embeddedKafka.consumeFromAnEmbeddedTopic(consumer, "test");
        ConsumerRecords<Integer, String> replies = KafkaTestUtils.getRecords(consumer);
        Assert.assertTrue(replies.count() == 1);
    }

}

However, you can also ignore the kafka processing annotations by excluding the KafkaAutoConfiguration in the test cases.

Note : This is applicable only if auto configuration is used. In case of custom configuration, ensure dependencies are loosely coupled to ignore the dependency injection issues.

AppTestWithoutKafka.java

@RunWith(SpringRunner.class)
@SpringBootTest
@EnableAutoConfiguration(exclude=KafkaAutoConfiguration.class)
public class AppTestWithoutKafka {

    @Test
    public void contextLoads() {
        System.out.println("context loads");
    }

}