如何訪問紗線簇中的Spark事件日志

Question

我正在群集模式下在紗線群集上運行spark。 我可以使用yarn logs命令訪問日志，但是在那兒看不到任何事件yarn logs 。 我想要這樣的東西在這里：

    {"Event":"SparkListenerTaskEnd","Stage ID":39,"Stage Attempt ID":0,"Task Type":"ShuffleMapTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":311,"Index":3,"Attempt":0,"Launch Time":1504799563465,"Executor ID":"driver","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1504799563539,"Failed":false,"Killed":false,"Accumulables":[{"ID":8079,"Name":"internal.metrics.executorDeserializeTime","Update":1,"Value":8,"Internal":true,"Count Failed Values":true},{"ID":8081,"Name":"internal.metrics.executorRunTime","Update":72,"Value":321,"Internal":true,"Count Failed Values":true},{"ID":8082,"Name":"internal.metrics.executorCpuTime","Update":15625000,"Value":218750000,"Internal":true,"Count Failed Values":true},{"ID":8083,"Name":"internal.metrics.resultSize","Update":2029,"Value":14203,"Internal":true,"Count Failed Values":true},{"ID":8086,"Name":"internal.metrics.memoryBytesSpilled","Update":0,"Value":0,"Internal":true,"Count Failed Values":true},{"ID":8087,"Name":"internal.metrics.diskBytesSpilled","Update":0,"Value":0,"Internal":true,"Count Failed Values":true},{"ID":8088,"Name":"internal.metrics.peakExecutionMemory","Update":3744,"Value":26208,"Internal":true,"Count Failed Values":true},{"ID":8096,"Name":"internal.metrics.shuffle.write.bytesWritten","Update":1047,"Value":7322,"Internal":true,"Count Failed Values":true},{"ID":8097,"Name":"internal.metrics.shuffle.write.recordsWritten","Update":1,"Value":7,"Internal":true,"Count Failed Values":true},{"ID":8098,"Name":"internal.metrics.shuffle.write.writeTime","Update":271872,"Value":7355654,"Internal":true,"Count Failed Values":true},{"ID":8099,"Name":"internal.metrics.input.bytesRead","Update":15377112,"Value":107190368,"Internal":true,"Count Failed Values":true},{"ID":8100,"Name":"internal.metrics.input.recordsRead","Update":105959,"Value":741704,"Internal":true,"Count Failed Values":true}]},"Task Metrics":{"Executor Deserialize Time":1,"Executor Deserialize CPU Time":0,"Executor Run Time":72,"Executor CPU Time":15625000,"Result Size":2029,"JVM GC Time":0,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":1047,"Shuffle Write Time":271872,"Shuffle Records Written":1},"Input Metrics":{"Bytes Read":15377112,"Records Read":105959},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerTaskStart","Stage ID":39,"Stage Attempt ID":0,"Task Info":{"Task ID":323,"Index":15,"Attempt":0,"Launch Time":1504799563539,"Executor ID":"driver","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerTaskEnd","Stage ID":39,"Stage Attempt ID":0,"Task Type":"ShuffleMapTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":317,"Index":9,"Attempt":0,"Launch Time":1504799563502,"Executor ID":"driver","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":1504799563540,"Failed":false,"Killed":false,"Accumulables":[{"ID":8079,"Name":"internal.metrics.executorDeserializeTime","Update":2,"Value":10,"Internal":true,"Count Failed Values":true},{"ID":8081,"Name":"internal.metrics.executorRunTime","Update":35,"Value":356,"Internal":true,"Count Failed Values":true},{"ID":8082,"Name":"internal.metrics.executorCpuTime","Update":31250000,"Value":250000000,"Internal":true,"Count Failed Values":true},{"ID":8083,"Name":"internal.metrics.resultSize","Update":2029,"Value":16232,"Internal":true,"Count Failed Values":true},{"ID":8086,"Name":"internal.metrics.memoryBytesSpilled","Update":0,"Value":0,"Internal":true,"Count Failed Values":true},{"ID":8087,"Name":"internal.metrics.diskBytesSpilled","Update":0,"Value":0,"Internal":true,"Count Failed Values":true},{"ID":8088,"Name":"internal.metrics.peakExecutionMemory","Update":3744,"Value":29952,"Internal":true,"Count Failed Values":true},{"ID":8096,"Name":"internal.metrics.shuffle.write.bytesWritten","Update":1045,"Value":8367,"Internal":true,"Count Failed Values":true},{"ID":8097,"Name":"internal.metrics.shuffle.write.recordsWritten","Update":1,"Value":8,"Internal":true,"Count Failed Values":true},{"ID":8098,"Name":"internal.metrics.shuffle.write.writeTime","Update":254977,"Value":7610631,"Internal":true,"Count Failed Values":true},{"ID":8099,"Name":"internal.metrics.input.bytesRead","Update":15521048,"Value":122711416,"Internal":true,"Count Failed Values":true},{"ID":8100,"Name":"internal.metrics.input.recordsRead","Update":105958,"Value":847662,"Internal":true,"Count Failed Values":true}]},"Task Metrics":{"Executor Deserialize Time":2,"Executor Deserialize CPU Time":0,"Executor Run Time":35,"Executor CPU Time":31250000,"Result Size":2029,"JVM GC Time":0,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Read Metrics":{"Remote Blocks Fetched":0,"Local Blocks Fetched":0,"Fetch Wait Time":0,"Remote Bytes Read":0,"Local Bytes Read":0,"Total Records Read":0},"Shuffle Write Metrics":{"Shuffle Bytes Written":1045,"Shuffle Write Time":254977,"Shuffle Records Written":1},"Input Metrics":{"Bytes Read":15521048,"Records Read":105958},"Output Metrics":{"Bytes Written":0,"Records Written":0},"Updated Blocks":[]}}
{"Event":"SparkListenerTaskStart","Stage ID":39,"Stage Attempt ID":0,"Task Info":{"Task ID":324,"Index":16,"Attempt":0,"Launch Time":1504799563546,"Executor ID":"driver","Host":"localhost","Locality":"PROCESS_LOCAL","Speculative":false,"Getting Result Time":0,"Finish Time":0,"Failed":false,"Killed":false,"Accumulables":[]}}
{"Event":"SparkListenerTaskEnd",

我如何找到它？ 更改event.logdir無效，有時會導致失敗。

Answer 1

好吧，我發現的唯一令人滿意的解決方案是在客戶端模式下運行，而不是在集群下運行以獲取事件日志...