Spring Kafka Test with Confluent Kafka Avro Serializer cannot find ZKClientConfig

Question

I struggle with starting embedded Kafka for intergration testing. This error is displayed:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig

This is my minimal sample. I need to use @ExtendWith(SpringExtension.class) as long as I need to autowire further Spring beans.

@ExtendWith(SpringExtension.class)
@EmbeddedKafka(
        partitions = 1,
        bootstrapServersProperty = "spring.kafka.bootstrap-servers")
public class KafkaTest {

    @Test
    public void test(EmbeddedKafkaBroker broker) {
        var brokerList = broker.getBrokersAsString();
        log.debug("Embedded Kakfa topics {}", broker.getTopics());
    }
}

I use:

• Java 11
• Spring Boot 2.3
• jUnit 5

My guess the dependency inconsistence takes the place here, however, it is weird since I believe Spring Boot parent can achieve the dependencies compatibility.

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.3.0.RELEASE</version>
    <relativePath/>
</parent>

For sure, I list related dependencies in pom.xml below:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>5.2.3</version>
</dependency>

<!-- TEST SCOPE -->
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.6.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>org.junit.vintage</groupId>
            <artifactId>junit-vintage-engine</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

I tried to exclude org.apache.kafka:kafka-clients in both org.springframework.kafka:spring-kafka-test and io.confluent:kafka-avro-serializer with no luck and the same error ocurred.

<exclusion>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
</exclusion>

Answer 1

Short answer:

These dependencies are in clash:

• io.confluent:kafka-avro-serializer:5.2.3
• org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE
• ... from the parent org.springframework.boot:spring-boot-starter-parent:2.3.0.RELEASE

The former one uses org.apache.zookeeper:zookeeper:jar:3.4.13:compile which should be excluded:

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>5.2.3</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Long answer:

Inspecing mvn dependency:tree before shows that kafka-avro-serializer includes surprisingly org.apache.zookeeper dependency while spring-kafka-test doesn't:

Extracted for kafka-avro-serializer

[INFO] +- io.confluent:kafka-avro-serializer:jar:5.2.3:compile  
[INFO] |  +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile    
[INFO] |  +- io.confluent:common-config:jar:5.2.3:compile   
[INFO] |  \- io.confluent:common-utils:jar:5.2.3:compile    
[INFO] |     +- org.apache.zookeeper:zookeeper:jar:3.4.13:compile   
[INFO] |     |  +- jline:jline:jar:0.9.94:compile   
[INFO] |     |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile  
[INFO] |     |  \- io.netty:netty:jar:3.10.6.Final:compile  
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile

Extracted for spring-kafka-test

[INFO] \- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test    
[INFO]    +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test 
[INFO]    +- org.apache.kafka:kafka-streams:jar:2.5.0:test  
[INFO]    |  +- org.apache.kafka:connect-json:jar:2.5.0:test    
[INFO]    |  |  \- org.apache.kafka:connect-api:jar:2.5.0:test  
[INFO]    |  \- org.rocksdb:rocksdbjni:jar:5.18.3:test  
[INFO]    +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test   
[INFO]    +- org.apache.kafka:kafka_2.12:jar:2.5.0:test 
[INFO]    |  +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test  
[INFO]    |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test    
[INFO]    |  |     \- com.thoughtworks.paranamer:paranamer:jar:2.8:test 
[INFO]    |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test 
[INFO]    |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test   
[INFO]    |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test  
[INFO]    |  +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test  
[INFO]    |  +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test   
[INFO]    |  +- org.scala-lang:scala-library:jar:2.12.10:test   
[INFO]    |  +- org.scala-lang:scala-reflect:jar:2.12.10:test   
[INFO]    |  +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test    
[INFO]    |  \- commons-cli:commons-cli:jar:1.4:test    
[INFO]    +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test    
[INFO]    \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test 
[INFO]       +- org.apiguardian:apiguardian-api:jar:1.1.0:test  
[INFO]       +- org.opentest4j:opentest4j:jar:1.2.0:test    
[INFO]       \- org.junit.platform:junit-platform-commons:jar:1.6.2:test

The spring-kafka-test is missing org.apache.zookeeper:zookeeper therefore the test cannot connect to Zookeeper and error is shown:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig

Excluding such dependency from kafka-avro-serializer fixes the problem (see the Short Answer ). The ouptut of mvn dependency:tree confirms the org.apache.zookeeper:zookeeper is properly excluded from kafka-avro-serializer and included in spring-kafka-test .

Extracted for kafka-avro-serializer :

+- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] |  +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] |  +- io.confluent:common-config:jar:5.2.3:compile
[INFO] |  \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile

Extracted for spring-kafka-test :

\- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO]    +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO]    +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO]    |  +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO]    |  |  \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO]    |  \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO]    +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO]    +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO]    |  +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO]    |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO]    |  |     \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO]    |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO]    |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO]    |  +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO]    |  +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO]    |  +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO]    |  +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO]    |  +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO]    |  +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO]    |  +- org.apache.zookeeper:zookeeper:jar:3.5.7:test
[INFO]    |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:test
[INFO]    |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO]    |  |  +- io.netty:netty-handler:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-common:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-resolver:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-buffer:jar:4.1.49.Final:test
[INFO]    |  |  |  +- io.netty:netty-transport:jar:4.1.49.Final:test
[INFO]    |  |  |  \- io.netty:netty-codec:jar:4.1.49.Final:test
[INFO]    |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.49.Final:test
[INFO]    |  |     \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:test
[INFO]    |  \- commons-cli:commons-cli:jar:1.4:test
[INFO]    +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO]    \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO]       +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO]       +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO]       \- org.junit.platform:junit-platform-commons:jar:1.6.2:test

Answer 2

I would like to add some information to the previous answer, because i ran into a similar problem today.

The main issue is that the kafka-client 2.5.0 is relying on Zookeeper 3.5 and no longer Zookeeper 3.4 (see https://spring.io/projects/spring-kafka for compatibiliy matrix)

The ZkClientConfig class has been introduced in Zookeeper 3.5.2 So any Zookeeper version below this will provoke this error. For testing purpose, you need to remove any zookeeper dependency already present and below 3.5.2 and add post 3.5.2 to your test dependencies.