View worker / executor logs in Spark UI since 1.0.0+

Question

In 0.9.0 to view worker logs it was simple, they where one click away from the spark ui home page.

Now (1.0.0+) I cannot find them. Furthermore the Spark UI stops working when my job crashes! This is annoying, what is the point of a debugging tool that only works when your application does not need debugging. According to http://apache-spark-user-list.1001560.n3.nabble.com/Viewing-web-UI-after-fact-td12023.html I need to find out what my master-url is, but I don't how to, spark doesn't spit out this information at startup, all it says is:

... -Dspark.master=\"yarn-client\" ...

and obviously http://yarn-client:8080 doesn't work. Some sites talk about how now in YARN finding logs has been super obfuscated - rather than just being on the UI, you have to login to the boxes to find them. Surely this is a massive regression and there has to be a simpler way??

How am I supposed to find out what the master URL is? How can I find my worker (now called executor) logs?

Answer 1

Depending on your configuration of YARN NodeManager log aggregation, the spark job logs are aggregated automatically. Runtime log is usually be found in following ways:

Spark Master Log

If you're running with yarn-cluster, go to YARN Scheduler web UI. You can find the Spark Master log there. Job description page "log' button gives the content.

With yarn-client, the driver runs in your spark-submit command. Then what you see is the driver log, if log4j.properties is configured to output in stderr or stdout.

Spark Executor Log

Search for "executorHostname" in driver logs. See comments for more detail.

Answer 2

These answers document how to find them from command line or UI

Where are logs in Spark on YARN?

For UI, on an edge node

Look in /etc/hadoop/conf/yarn-site.xml for the yarn resource manager URI ( yarn.resourcemanager.webapp.address ).

Or use command line:

yarn logs -applicationId [OPTIONS]