Spark having different logs for each task in worker
Question
Is it possible to have different logs file for task in a single worker? Right now I get all the logs in the same file (for different tasks) in the worker.
I am using spark-standalone and the log directory is in SPARK_HOME/work
1 answers
solution1
0 2016-02-24 15:40:41
You can configure SPARK_WORKER_DIR for each individual worker via the spark-env.sh configuration file. By changing this setting it will change where the logs are output by default.
See the deployment documentation for more information.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Each spark worker only having 2 executors
Spark SparkOutOfMemoryError despite having each task memory greater than input
Exposing Spark Worker (stdout stderr) logs in Kubernetes
Spark structured streaming process each row on different worker nodes as soon as it arraives
Meaning of the task id format in Spark logs
What is a task in Spark? How does the Spark worker execute the jar file?
How to get the worker logs in spark using the default cluster manager?
Exposing Spark Driver and Worker stdout stderr logs in Kubernetes to History Server
View worker / executor logs in Spark UI since 1.0.0+
Spark worker node cannot start. Incorrect java path in logs
Related Tags