Checking yarn application logs

Question

I am new to spark. I have a 10node Hadoop cluster with one edge node. I am submitting spark application from edge node and redirecting spark-submit command output to local file on edge node.

So when spark application fails I can check edge node log file and take an action.

When I read about yarn application logs,it is said that node managers running that application will log into some location (yarn.nodemanager.log-dir).

How is this nodemanager log different from edge node log. Can anyone explain yarn application logs in detail.

Answer 1

"Edge node logs" would be Spark driver application logs, which would likely say something like URL to track the Job: <link to YARN UI>

If you want the actual Spark runtime logs, you need to look at the inidivual Spark executors via the Spark UI (which redirect to the YARN UI, if that is how you run Spark)

The NodeManager (and ResourceManager) is a YARN process, with its own logs, and not related to your Spark code