![](/img/trans.png)
[英]Spark Streaming + Kafka: how to check name of topic from kafka message
[英]How to read records from Kafka topic from beginning in Spark Streaming?
我正在尝试使用Spark Streaming从Kafka主题中读取记录。
这是我的代码:
object KafkaConsumer {
import ApplicationContext._
def main(args: Array[String]) = {
val kafkaParams = Map[String, Object](
"bootstrap.servers" -> "localhost:9092",
"key.deserializer" -> classOf[StringDeserializer],
"value.deserializer" -> classOf[StringDeserializer],
"group.id" -> s"${UUID.randomUUID().toString}",
"auto.offset.reset" -> "earliest",
"enable.auto.commit" -> (false: java.lang.Boolean)
)
val topics = Array("pressure")
val stream = KafkaUtils.createDirectStream[String, String](
streamingContext,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
stream.print()
stream.map(record => (record.key, record.value)).count().print()
streamingContext.start()
}
}
我运行时没有显示任何内容。
为了检查pressure
主题中是否确实存在数据,我使用命令行方法并显示记录:
bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic pressure \
--from-beginning
输出:
TimeStamp:07/13/16 15:20:45:226769,{'Pressure':'834'}
TimeStamp:07/13/16 15:20:45:266287,{'Pressure':'855'}
TimeStamp:07/13/16 15:20:45:305694,{'Pressure':'837'}
怎么了?
你缺少streamingContext.awaitTermination()
。
您需要启动streamingContext
并最终执行streamingContext.awaitTermination()
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.