How to solve Kafka message producer NoSuchMethodError org.apache.kafka.common.security.ssl.SslFactory.sslEngineBuilder()

Question

I'm trying to run a simple message producer that leverages SSL security to connect with Confluent platform. I am using the KafkaTemplate from Spring Kafka for sending the message. However, at the time I call kafkaTemplate.send() , I receive the following message:

Failed to construct kafka producer
java.lang.NoSuchMethodError: org.apache.kafka.common.security.ssl.SslFactory.sslEngineBuilder()Lorg/apache/kafka/common/security/ssl/SslEngineBuilder;

I am not sure if this issue stems from a missing property in the properties file (eg spring.kafka.properties.* ), or if the issue is one of dependencies. In my pom.xml file I have the following:

     <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>5.5.1</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-streams-avro-serde</artifactId>
            <version>5.5.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-client-plugins</artifactId>
            <version>5.5.1-ce</version>
        </dependency>
    </dependencies>

Up to now I have been unable to find why that particular method isn't being found. I suspect I am missing one or more properties, but I'm unsure which one(s).

Within the properties file I have the following values

spring.kafka.properties.basic.auth.credentials.source=USER_INFO
spring.kafka.properties.security.protocol=SASL_SSL
spring.kafka.properties.sasl.mechanism=OAUTHBEARER
spring.kafka.properties.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required username=<removed> password=<removed> metadataServerUrls=<removed>

Answer 1

Based on the research I've done and the help provided here, I've been able to determine the problem is one of compatibility. The Confluent 5.5.1 libraries are incompatible with Spring Boot 2.4.x, because the kafka-clients 2.6.0 that is compatible with Spring Boot 2.4.x is not compatible with that version of Confluent.