Apache Spark executor uses more than one core despite of spark.executor.cores=1

Question

When I start Apache Spark 1.2.1 application on CentOS 6.5, I receive more than 100% load for executors in accordance to 'top' output and load average is significant more than amount of cores.

As a result I have high load on garbage collector.

• Have tried to limit cores per executor with spark.executor.cores=1 .
• Have tried spark.cores . No any effect.
• Hardware is 2 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 6 physical cores each so 12 CPU cores per node.
• Deployment model is YARN client.

Similar Ubuntu 14.04 setup with 4 physical cores (Intel i5) has no any issue, 1 core per executor.

Any idea how to fix this?

Application submission is performed from code with all needed properties set through System.setProperty and then Spark configuration and context created. It is done the same way, the only possible difference could be Spark configuration properties set which is per-cluster but there is nothing special. Under Ubuntu with 4 cores i5 this leads to proper load with no more than 1 core used by each executor. Under CentOS 6.5 with 2x6 cores E5 I see more than one core used per executor. More, I tried to apply 4 cores i5 configuration to E5 and had no success.

spark-defaults.conf file content (before spark version substitution which is currently 1.2.1):

spark.master=yarn-client
spark.eventLog.enabled=true
spark.eventLog.dir=hdfs:///user/spark/applicationHistory
spark.yarn.historyServer.address=X.X.X.X:18088
spark.executor.memory=1650M
spark.executor.cores=1
spark.cores.max=4
spark.executor.instances=15
spark.shuffle.memoryFraction=0.2
spark.storage.memoryFraction=0.02
spark.yarn.jar=hdfs:///user/spark/share/lib/${spark.version}/spark-assembly.jar

Main problem here is I see 2 x 6 cores E5 has lower performance than linear 1 i5 x 4 cores. Yes, E5 is somewhat older but should be anyway notable more powerful. And yet analysed Spark History server UI on similar load on both clusters I see notable more time spent into GC on E5 cluster. Crazy state.

Answer 1

OK, at the end I have found:

1. Actually I had just wrong understanding of 'CPU load' here. More than 100% means only execution queue length.
2. CPU cores allocation was correct. 1 core was allocated per executor which was demanded.
3. Based on GC profiling enabled on Spark executors and then job code review it was found root cause was just non-optimal copying of some objects. Code was corrected to be more lean in term of memory management.

So the resolution was:

When you see more than 100% load on CPU core under Spark executor you should first check your job GC logs for ineffective memory intensive operations or at least reduce lifetime of some objects by more aggressive release of resources.

Answer 2

You're requesting more executors than you have CPU cores, which, if you think about it, should be impossible. However the default YARN model is to consider only RAM as a limiting factor (using the DefaultResourceCalculator ), and it will happily share CPU cores between multiple "1 core" executor, effectively leading to a load superior to 1 on CPU cores. You can use the DominantresourceCalculator to avoid this :)