how to get aggregated logs from running Spark application
Question
I have a continuous running Spark Application which reads msgs from Kafka and does some processing. Is there a way to get aggregated "Application Logs" ?
AFAIK, log aggregation will happen only when SparkContext is destroyed.
1 answers
solution1
0 2016-10-05 16:35:55
Have you tried standard YARN command for logs?
yarn logs -applicationId some-id
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How can I see the aggregated logs for a Spark standalone cluster
Separating application logs in Logback from Spark Logs in log4j
Running Spark Application from Eclipse
How to retain Spark executor logs in Yarn after Spark application is crashed
Is it possible to get sparkcontext of an already running spark application?
How to kill apache spark application running in background after killing it from SPARK web UI
Running spark application from HDInsight cluster headnode
I am using spark for my application, but i am getting unnecessary logs. How can i disable logs in spark java application
How to Stop running Spark Streaming application Gracefully?
How to get the worker logs in spark using the default cluster manager?
Related Tags