Spark History UI not working | Ambari | YARN

Question

I have a hadoop cluster setup using Ambari which has services like HDFS,YARN,spark running on the hosts. When i run the sample spark pi in cluster mode as master yarn, the application gets successfully executed and I can view the same from resource manager logs.

But when i click on the history link, it does not show the spark history UI. How to enable/view the same?

Answer 1

First, check if your spark-history server is already configured by looking for spark.yarn.historyServer.address in spark-defaults.conf file.

If not configured, this link should help you configure the server: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_installing_manually_book/content/ch19s04s01.html

If already configured, check if the history server host is accessible from all the nodes in the cluster, and also the port is open.