将 Spark 指标推送到 Prometheus

Question

How can I set Spark to push metrics to Prometheus? like it sends to Graphite for example.

Of course, I don't care about using the Prometheus push getaway but how do I get Spark to send those metrics (and not just expose them at an endpoint)?

Answer 1

BanzaiCloud have an implementation of Prometheus push gateway sink for spark metrics You can view the full guide here: https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md EDIT: The current build is for Spark 2.XI never tried it on Spark 3.X but I it could works just fine since I dont recall any major changes in the metrics system

Basically you need to do 2 thing:

1. set the spark metrics sink to the push gateway server metrics.properties (copied the snippet from the guide):

 # Enable Prometheus for all instances by class name
*.sink.prometheus.class=org.apache.spark.banzaicloud.metrics.sink.PrometheusSink
# Prometheus pushgateway address
*.sink.prometheus.pushgateway-address-protocol=<prometheus pushgateway protocol> - defaults to http
*.sink.prometheus.pushgateway-address=<prometheus pushgateway address> - defaults to 127.0.0.1:9091
*.sink.prometheus.period=<period> - defaults to 10
*.sink.prometheus.unit=< unit> - defaults to seconds (TimeUnit.SECONDS)
*.sink.prometheus.pushgateway-enable-timestamp=<enable/disable metrics timestamp> - defaults to false

# There are more options in the guide

2. Add their spark-metris jar to your app using --jars <path to jar> (by downloading it from maven central) or --packages <groupId:artifactId:version> (which pulls it from maven centeral or other custom repo using --repositories <url> ) in the spark-submit command also add the metrics.properties file to the deployment using --files /path/to/metrics.properties

# Using packages
spark-submit ... --packages com.banzaicloud:spark-metrics_2.12:2.4-1.0.6 --files /path/to/metrics.properties ...

# Using JAR
spark-submit ... --jars /path/to/spark-metrics_2.12-2.4-1.0.6.jar --files /path/to/metrics.properties ...