使用Prometheus JMX代理从在单个节点上运行的多个Spark执行者导出指标

Question

I am running spark cluster and I have one node which has three executors running on it. I want to scrap metrics for all three executors using Prometheus JMX agent. I am passing Prometheus java agent using "spark.executor.extraJavaOptions" in spark submit command like below.

--conf "spark.executor.extraJavaOptions=-javaagent:/opt/agent/jmx_prometheus_javaagent-0.3.1.jar=6677:/opt/agent/spark.yml" 

I am passing port as 6677 and JMX metrics are available for one executor only. For other two executors javaagent will fail as port 6677 is already in use and there will be no metrics reported for other two executors. Can someone please guide me how to solve this problem. I found similar question here but there is no answer for that.

Answer 1

Use different ports for the other 2 executors? You can't have 3 servers listening on the same port and this has nothing to do with Prometheus or JMX.