將 Spark 指標推送到 Prometheus

Question

如何設置 Spark 以將指標推送到 Prometheus？ 例如，它發送到 Graphite。

當然，我不關心使用 Prometheus push getaway，但我如何讓 Spark 發送這些指標（而不僅僅是在端點上公開它們）？

Answer 1

BanzaiCloud 實現了用於 Spark 指標的 Prometheus 推送網關接收器您可以在此處查看完整指南： https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md編輯：當前版本適用於 Spark 2 .XI 從未在 Spark 3.X 上嘗試過，但我認為它可以正常工作，因為我不記得指標系統有任何重大變化

基本上你需要做兩件事：

1. 將 spark 指標接收器設置為推送網關服務器 metrics.properties（從指南復制片段）：

 # Enable Prometheus for all instances by class name
*.sink.prometheus.class=org.apache.spark.banzaicloud.metrics.sink.PrometheusSink
# Prometheus pushgateway address
*.sink.prometheus.pushgateway-address-protocol=<prometheus pushgateway protocol> - defaults to http
*.sink.prometheus.pushgateway-address=<prometheus pushgateway address> - defaults to 127.0.0.1:9091
*.sink.prometheus.period=<period> - defaults to 10
*.sink.prometheus.unit=< unit> - defaults to seconds (TimeUnit.SECONDS)
*.sink.prometheus.pushgateway-enable-timestamp=<enable/disable metrics timestamp> - defaults to false

# There are more options in the guide

2. Add their spark-metris jar to your app using --jars <path to jar> (by downloading it from maven central) or --packages <groupId:artifactId:version> (which pulls it from maven centeral or other custom repo using --repositories <url> ）在 spark-submit 命令中還使用--files /path/to/metrics.properties將 metrics.properties 文件添加到部署中

# Using packages
spark-submit ... --packages com.banzaicloud:spark-metrics_2.12:2.4-1.0.6 --files /path/to/metrics.properties ...

# Using JAR
spark-submit ... --jars /path/to/spark-metrics_2.12-2.4-1.0.6.jar --files /path/to/metrics.properties ...