如何在 Kubernetes 中将 Spark 的 JVM 指标发送到 Prometheus

Question

I am using the Spark operator to run Spark on Kubernetes. ( https://github.com/GoogleCloudPlatform/spark-on-k8s-operator )

I am trying to run a Java agent in Spark driver and executor pods and send the metrics through a Kubernetes service to Prometheus operator.

I am using this example https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi-prometheus.yaml

Java agent is exposing the metrics on port 8090 for a short time (I can validate that with port-forwarding kubctl port-forward < spark-driver-pod-name > 8090:8090 ), also the service is also exposing the metrics for few mins ( can validate that with port-forwarding kubctl port-forward svc/< spark-service-name > 8090:8090 ).

Promethues is able to register these pod's URL in the prometheus, but when it is trying to scrape the metrics(runs for every 30 seconds), the pod's URL is down.

How can i make the Java agent JMX exporter to run long, until the driver and executors completed the job. could you please guide or help me here, who have come across this scenario before?

Answer 1

Either Prometheus needs to scrape the metrics of every 5 seconds (chances are that metrics may not be accurate), or you need to use pushgateway, like mentioned in this blog( https://banzaicloud.com/blog/spark-monitoring/ ) to push the metrics to Prometheus

Pushing the metrics to Prometheus , is a best practice for batch jobs . Pulling the metrics from Prometheus is a best approach for long running services (ex:REST Services)