Spark 2.3.1 on YARN : how to monitor stages progress programatically?

Question

I have a setup with Spark running on YARN, and my goal is to programmatically get updates of the progress of a Spark job by its application id.

My first idea was to parse HTML output of the YARN GUI. However, the problem of such GUI, is that the progress bar associated to a spark job don't get updated regularly and even don't change most of time : when the job start, the percent is something like 10%, and it stuck to this value until the job finish. So such YARN progress bar is just irrelevant for Spark Jobs.

When I click to the Application Master link corresponding to a Spark Job, I'm redirected to the Spark GUI that is temporarily binded during the job run. The stages page is very relevant about progress of the Spark job. However it is plain HTML, so it is a pain to parse. On the Spark documentation, they talk about a JSON API, however it seems that I can't access to it as I'm under YARN and I'm accessing Spark GUI trough YARN proxy pages.

May be a solution, in order to have access to more things, could be to access to the real Spark GUI ip:port, and not the YARN proxied one, but I don't know if I can get such source URL easily...

All of that sound complicated to just get Spark job progress... As of 2018, could you please tell us what are the preferred methods to get relevant stages progress of a Spark Job running on YARN ?

Answer 1

You can retrieve YARN application state and other details for your submitted spark on yarn job via REST API

Refer to the below links:

https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html#Example_usage

https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_API

Answer 2

从应用程序内部，您可以使用spark.sparkContext.statusTracker获取有关舞台进度的信息，您可以查看Zeppelin Notebook如何为Spark 2.3实现进度条： https : //github.com/apache/zeppelin/blob/主/火花/火花-scala-parent / src / main / scala / org / apache / zeppelin / spark / JobProgressUtil.scala

Answer 3

No way of knowing the progress in percentage, as you can have any amount of Spark stages. However, there is a REST API for Spark History Server - Monitoring and Instrumentation with which you can ask for stages/tasks/jobs info. Assuming your app has predefined amount of Stages - you can calculate the progress.