How to access only spark logs in a yarn cluster

Question

I'm using spark with yarn in cluster mode and I wouldlike to retrieve only the logs concerning spark in order to use the history server. Is there a way to do that instead of copy pasting it from the big log files ?

Answer 1

Yes, you can take logs of each task. You need to go to the yarn UI (8088 port) and copy id of the task. Then use command:

yarn logs --applicationId <TASK_ID> 

For example:

yarn logs --applicationId application_1504771895665_0001    

By this command you can see logs of each container that task used.