How to check the number of cores Spark uses?
Question
I have spark.cores.max set to 24 [3 worker nodes], but If I get inside my worker node and see there is just one process [command = Java] running that consumes memory and CPU. I suspect it does not use all 8 cores (on m2.4x large ).
How to know the number?
1 answers
solution1
3 2014-09-10 21:44:29
You can see the number of cores occupied on each worker in the cluster under the Spark Web UI:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
How to change number of cores in spark?
How to check how many cores PySpark uses?
How to set spark to use a specific number of cores?
How to know number of cores utilized by a stage in spark
Spark number of cores used
How to get number of executors and number of cores in Java spark
How to determine number of partitons of rdd in spark given the number of cores and executors ?
Spark number of cores used by driver
Spark Resource Allocation: Number of Cores
How to tune spark executor number, cores and executor memory?
Related Tags