[英]scala kafka producer:Can't read Kafka version from MANIFEST.MF. Possible cause: java.lang.NullPointerException
Part of build.sbt build.sbt的一部分
scalaVersion := "2.11.0"
val sparkVersion = "2.1.1";
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion % "provided";
libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided";
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.2";
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % sparkVersion;
Producer code: 生产者代码:
updates.foreachPartition{ partition =>
val kafkaProperties = new Properties();
kafkaProperties.put("serializer.class","kafka.serializer.StringEncoder");
kafkaProperties.put("metadata.broker.list",kafkaBrokers);
//kafkaProperties.put("partitioner.class","RuleUpdateKafkaPartitioner");
val kafkaConfig = new ProducerConfig(kafkaProperties);
val kafkaProducer = new Producer[String,String](kafkaConfig);
partition.foreach { record =>
...
val data = new KeyedMessage[String,String](kafkaTopic,record.ruleID(),record_json.toString());
kafkaProducer.send(data);
}
kafkaProducer.close();
Error: 错误:
17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 WARN AppInfo$: Can't read Kafka version from MANIFEST.MF.
17/06/16 17:52:05 INFO VerifiableProperties:验证属性17/06/16 17:52:05 INFO VerifiableProperties:属性meta.broker.list被覆盖到10.1.29.186:9092 17/06/16 17:52 :05信息VerifiableProperties:属性serializer.class被覆盖到kafka.serializer.StringEncoder 17/06/16 17:52:05 WARN AppInfo $:无法从MANIFEST.MF中读取Kafka版本。 Possible cause: java.lang.NullPointerException 17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO VerifiableProperties: Verifying properties 17/06/16 17:52:05 INFO VerifiableProperties: Property metadata.broker.list is overridden to 10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties: Property serializer.class is overridden to kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO Producer: Shutting down producer
可能的原因:java.lang.NullPointerException 17/06/16 17:52:05 INFO VerifiableProperties:验证属性17/06/16 17:52:05 INFO VerifiableProperties:属性metadata.broker.list被覆盖到10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties:属性serializer.class被覆盖到kafka.serializer.StringEncoder 17/06/16 17:52:05 INFO VerifiableProperties:验证属性17/06/16 17:52:05 INFO VerifiableProperties:属性meta.broker.list被覆盖到10.1.29.186:9092 17/06/16 17:52:05 INFO VerifiableProperties:属性serializer.class被覆盖到kafka.serializer.StringEncoder 17/06/16 17:52 :05 INFO制作人:关闭制作人
UPDATE kafka is kafka_2.11-0.10.2.0 更新 kafka是kafka_2.11-0.10.2.0
This is a warning and your application won't crash because of it. 这是一个警告,您的应用程序不会因此而崩溃。 The codes here are trying to load Kafka version from
MANIFEST.MF
. 此处的代码正在尝试从
MANIFEST.MF
加载Kafka版本。 My hunch is you use a shaded jar and it doesn't include Kafka's MANIFEST.MF
. 我的直觉是您使用有阴影的罐子,但其中不包括Kafka的
MANIFEST.MF
。
See https://issues.apache.org/jira/browse/KAFKA-1901 for more details. 有关更多详细信息,请参见https://issues.apache.org/jira/browse/KAFKA-1901 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.