how to find out driver process node for tasks running in spark in yarn-cluster mode

Question

My spark is running in yarn cluster mode .When I submit a job , I want to find out which machine does the driver program locates in my cluster. It seems that I can only get the container location from the yarn web ui:

What can I do?

Answer 1

The Spark Web UI is launched on port 4040 of the driver node, so one way to find the driver's ip is to click on the logs link and search for 4040

A sample log line would be SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.100.0.117:4040 where 10.100.0.117 is the ip address of the driver node.