Kafka Producer Class Not Found Exception

Question

I tried to implement a simple producer consumer example with kafka and I achieved with the following properties:

Properties configProperties = new Properties();
    configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:" + portNumber);
    configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.ByteArraySerializer");
    configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");

    // Belirtilen property ayarlarına sahip kafka producer oluşturulur
    org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties);

However when I try the exact same configs, and everything else the same, in another project which is a plugin for a data visualization software, I got this error:

.... // Here there is some other stuff but I thing the important one is the below one
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Producer
at App.MyControlPanel.<init>(MyControlPanel.java:130)
at App.CytoVisProject.<init>(CytoVisProject.java:29)
... 96 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.producer.Producer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 98 more

In the first one that I said it worked, I was using "mvn clean compile assembly:single", but in the second one I created a jar file for the whole project. Because the visualization software wants a jar file to install the plugin. Since every thing is same (At least I could not find any difference, I used same code) I guess the problem is about the way build the project. What happened here? What is the difference between "mvn clean compile assembly:single" and building a jar file in IntelliJ? Why I got this error and how to fix this? Thanks a lot for help!

---

As I said in the last comment of the first answer, I have a plugin which has manifest and transform as goal. Here:

<plugin>
            <groupId>com.springsource.bundlor</groupId>
            <artifactId>com.springsource.bundlor.maven</artifactId>
            <version>1.0.0.M2</version>
            <configuration>
                <outputManifest>C:\Users\USER\AppData\Local\Temp\archetype2tmp/META-INF/MANIFEST.MF</outputManifest>
                <failOnWarnings>false</failOnWarnings>
                <removeNullHeaders>true</removeNullHeaders>
                <manifestHeaders><![CDATA[Bundle-ManifestVersion: 2
Bundle-Name: CytoVisProject
Bundle-SymbolicName: CytoVisProject
Spring-DM-Version: ${pom.version}
]]></manifestHeaders>
            </configuration>
            <!-- generate the manifest automatically during packaging -->
            <executions>
                <execution>
                    <id>bundle-manifest</id>
                    <phase>package</phase>
                    <goals>
                        <goal>manifest</goal>
                        <goal>transform</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

If I use a shade plugin like below:

<plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>

            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

It does not works, because I need to use manifest and transform as goals in my plugin. How can I add kafka classes to the jar file that IntelliJ creates to solve this problem (I am not sure if this can solve or not)?

Answer 1

I've found an easier solution. I have changed

kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");

to this

kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

And afterwards my code run, also as part of a pre-compiled plug-in.

Answer 2

What the problem is

At compile time kafka-clients-0.10.0.0.jar is available, so the code compiles successfully, but at runtime it is missing, hence the error you get.

How to fix it

You have 2 options:

• Include the kafka JAR inside your JAR
• Add the kafka JAR to your plugin's classpath

Answer 3

另外，请确保您没有对标有提供范围的 kafka 客户端的依赖项。