[英]How to do writeStream a dataframe in console? (Scala Spark Streaming)
I want to debug my notebook thus I need to print out the streaming-data in notebook console mode.我想调试我的笔记本,因此我需要在笔记本控制台模式下打印出流数据。 I have two questions: 1- Is it possible to do:我有两个问题:1-是否可以这样做:
df.writeStream.format("console").start().awaitTermination()
2- If yes, where can I see the output? 2- 如果是,我在哪里可以看到 output?
This is after 10 minutes... No error or result这是 10 分钟后...没有错误或结果
Thanks!谢谢!
I figured out what the problem in my case was.我弄清楚我的问题是什么。 I had to add the ".outputMode("append")" in my method.我必须在我的方法中添加“.outputMode(“append”)” 。 Here is how it looks:这是它的样子:
def writeStreamData(dataFrame: DataFrame): Unit = {
/**
* write the given dataframe into a file or console
:params: dataframe
*/
dataFrame.writeStream
.format("console")
.outputMode("append")
.start()
.awaitTermination()
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.