独立集群模式：spark如何分配spark.executor.cores？

Question

I'm searching for how and where spark allocates cores per executor in the source code. Is it possible to control programmaticaly allocated cores in standalone cluster mode?

Regards, Matteo

Answer 1

Spark allows for configuration options to be passed through the .set method on the SparkConf class.

Here's some scala code that sets up a new spark configuration:

new SparkConf()
  .setAppName("App Name")
  .setMaster('local[2]')
  .set("spark.executor.cores", "2")

Documentation about the different configuration options:

http://spark.apache.org/docs/1.6.1/configuration.html#execution-behavior

I haven't looked through the source code exhaustively, but I think this is the spot in the source code where the executor cores are defined prior to allocation:

https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/main/scala/org/apache/spark/scheduler/cluster/ExecutorData.scala

Answer 2

In stand alone mode, you have following options:

a. While starting the cluster, you can mention how many cpu cores to be allotted for spark applications. This can be set both as env variable SPARK_WORKER_CORES or passed as argument to shell script (-c or --cores)

b. Care should be taken (if other applications also share resources like cores) not to allow spark to take all the cores. This can be set using spark.cores.max parameter.

c. You can also pass --total-executor-cores <numCores> to the spark shell

For more info, you can look here