[英]Streaming tab is not showing for structured streaming
I am using structured streaming for reading csvs and writing to kafka. 我正在使用结构化流媒体来阅读csvs并写入kafka。 The streaming tab is not showing in Spark UI (not using streaming context).
流媒体选项卡未显示在Spark UI中(不使用流式上下文)。
val userSchema = new StructType().add("name", "string").add("age", "integer")
val csvDF = spark
.readStream
.option("sep", ";")
.schema(userSchema) // Specify schema of the csv files
.csv("/path/to/directory")
How can I get streaming metrics in the UI? 如何在UI中获取流式传输指标?
To see some metrics (in console), you need to add a listener 要查看某些指标(在控制台中),您需要添加一个监听器
spark.streams.addListener(new StreamingQueryListener {
override def onQueryStarted(event: StreamingQueryListener.QueryStartedEvent): Unit = logger.debug(s"QueryStarted [id = ${event.id}, name = ${event.name}, runId = ${event.runId}]")
override def onQueryProgress(event: StreamingQueryListener.QueryProgressEvent): Unit = logger.warn(s"QueryProgress ${event.progress}")
override def onQueryTerminated(event: StreamingQueryListener.QueryTerminatedEvent): Unit = logger.debug(s"QueryTerminated [id = ${event.id}, runId = ${event.runId}, error = ${event.exception}]")
})
QueryProgressEvent
, displays info about offset, watermarks, source, sinks,etc. QueryProgressEvent
,显示有关偏移,水印,源,接收器等的信息。
This video can help you : Monitoring Structured Streaming Applications 此视频可以帮助您: 监控结构化流媒体应用程序
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.