HDP平台上的Kafka NoSuchMethodError

Question

Case

Trying to build some code which requires Kafka libraries and will be deployed to an HDP platform. Build succeeds but at runtime complains that classes cannot be found. Looking up Kafka version on HDP platform reveals something like 0.10.0.2.5.3.16-1 , a quick check on mvn reveals that no such Kafka library exists for this version. The nearest version is 0.10.0.2.5.3.42-1 which I'm using.

Symptoms

Runtime errors which say that classes or methods cannot be found, for instance kafka.javaapi.consumer.SimpleConsumer.<init>(Ljava/lang/String;IIILjava/lang/String;Ljava/lang/String;)V

Answer 1

Reason

The services provided in the HDP platform are suffixed with the HDP version, so Kafka 0.10.0.2.5.3.16-1 is actually Kafka version 0.10.0 , and digging further reveals that it's version 0.10.0.1 . The HDP version is 2.5.3 .

Solution

Drop back ${kafka.version} in pom to a version which works, I found that 0.8.2.1 worked for me.