scala kafka producer:Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException

Question

Part of build.sbt

scalaVersion := "2.11.0"

val sparkVersion = "2.1.1";

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided";

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.2";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % sparkVersion;

Producer code:

        updates.foreachPartition{ partition =>
            val kafkaProperties = new Properties();
            kafkaProperties.put("serializer.class","kafka.serializer.StringEncoder");
            kafkaProperties.put("metadata.broker.list",kafkaBrokers);
            //kafkaProperties.put("partitioner.class","RuleUpdateKafkaPartitioner");
            val kafkaConfig = new ProducerConfig(kafkaProperties);
            val kafkaProducer = new Producer[String,String](kafkaConfig);
            partition.foreach { record =>
                ...
                val data = new KeyedMessage[String,String](kafkaTopic,record.ruleID(),record_json.toString());
                kafkaProducer.send(data);
            }
            kafkaProducer.close();

Error:

17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 WARN AppInfo$: Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException 17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO Producer: Shutting down producer

UPDATE kafka is kafka_2.11-0.10.2.0

Answer 1

This is a warning and your application won't crash because of it. The codes here are trying to load Kafka version from MANIFEST.MF . My hunch is you use a shaded jar and it doesn't include Kafka's MANIFEST.MF .

See https://issues.apache.org/jira/browse/KAFKA-1901 for more details.