How to know number of cores utilized by a stage in spark
Question
How to know number of cores utilized by a stage in spark. Shall we treat number of tasks in a stage is equal to number of cores. Since, each task runs on a separate core.
1 answers
solution1
1 2019-11-27 13:49:59
Spark UI is the easiest possible way.Navigate to "Environment" tab in Spark UI. From YARN, click on the application Id & then on the Tracking URL. It directs to the Spark UI page
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to change number of cores in spark?
How to set spark to use a specific number of cores?
How to check the number of cores Spark uses?
Spark number of cores used
can someone let me know how to decide --executor memory and --num-of-executors in spark submit job . What is the concept of -number-of-cores
How to get number of executors and number of cores in Java spark
How to determine number of partitons of rdd in spark given the number of cores and executors ?
Spark number of cores used by driver
How to tune spark executor number, cores and executor memory?
How to know which stage of a job is currently running in Apache Spark?
Related Tags