简体   繁体   中英

What do these metrics mean for Spark Structured Streaming?

spark.streams.addListener(new StreamingQueryListener() {
    ......
    override def onQueryProgress(queryProgress: QueryProgressEvent): Unit = {
        println("Query made progress: " + queryProgress.progress)
    }
    ......
})

When StreamingQueryListener is added to Spark Structured Streaming session and output the queryProgress continuously, one of the metrics you will get is durationMs :

Query made progress: {
  ......
  "durationMs" : {
    "addBatch" : 159136,
    "getBatch" : 0,
    "getEndOffset" : 0,
    "queryPlanning" : 38,
    "setOffsetRange" : 14,
    "triggerExecution" : 159518,
    "walCommit" : 182
  }
  ......
}​

Can anyone told me what do those sub-metrics in durationMs meaning in spark context? For example, what is the meaning of "addBatch 159136".

https://www.waitingforcode.com/apache-spark-structured-streaming/query-metrics-apache-spark-structured-streaming/read

This is an excellent site that addresses the aspects and more, passing the credit to this site therefore.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM