测试 Flink-Kafka 集成

Question

How to set up a unit testing environment where I can create Kafka topics, write to them using FlinkKafkaProducer and read from them using FlinkKafkaConsumer?

It seems like the functionality is there, but is not well-documented.

The farthest I was able to get is the following

val server = new KafkaTestEnvironmentImpl()
server.prepare(KafkaTestEnvironment.createConfig().setKafkaServersNumber(1))
server.createTestTopic("my.test.topic", 1, 1)
server.shutdown()

But this throws an error java.lang.NoClassDefFoundError: kafka/admin/RackAwareMode . I am not sure if trying to resolve this error is the best course of action or whether there is a simpler solution.

Answer 1

I'd highly recommend to user a docker-based approach, such as testcontainers . Setup of Kafka is as easy as it gets.

If you need to inject data, there is another framework that allows you to maintain the topics and add data easily. If can be used in conjunction with testcontainers or if you really only need Kafka as an external system it can even fully replace it.