Spark Standalone: application gets 0 cores

Question

I seem to be unable to assign cores to an application. This leads to the following (apparently common) error message:

Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

I have one master and two slaves in a Spark cluster. All are 8-core i7s with 16GB of RAM.

I have left the spark-env.sh virtually virgin on all three, just specifying the master's IP address.

My spark-submit is the following:

nohup ./bin/spark-submit
  --jars ./ikoda/extrajars/ikoda_assembled_ml_nlp.jar,./ikoda/extrajars/stanford-corenlp-3.8.0.jar,./ikoda/extrajars/stanford-parser-3.8.0.jar \
  --packages datastax:spark-cassandra-connector:2.0.1-s_2.11 \
  --class ikoda.mlserver.Application \
  --conf spark.cassandra.connection.host=192.168.0.33 \
  --conf spark.cores.max=4 \
  --driver-memory 4g –num-executors 2 --executor-memory 2g --executor-cores 2 \
  --master spark://192.168.0.141:7077 ./ikoda/ikodaanalysis-mlserver-0.1.0.jar 1000 > ./logs/nohup.out &

I suspect I am conflating the sparkConf initialization in my code with the spark-submit . I need this as the app involves SparkStreaming which can require reinitializing the SparkContext .

The sparkConf setup is as follows:

val conf = new SparkConf().setMaster(s"spark://$sparkmaster:7077").setAppName("MLPCURLModelGenerationDataStream")

conf.set("spark.streaming.stopGracefullyOnShutdown", "true")
conf.set("spark.cassandra.connection.host", sparkcassandraconnectionhost)
conf.set("spark.driver.maxResultSize", sparkdrivermaxResultSize)
conf.set("spark.network.timeout", sparknetworktimeout)
conf.set("spark.jars.packages", "datastax:spark-cassandra-connector:"+datastaxpackageversion)
 conf.set("spark.cores.max", sparkcoresmax)

The Spark UI shows the following:

Answer 1

OK, this is definitely a case of programmer error.

But maybe others will make a similar error. The Master had been used as a local Spark previously. I had put some executor settings in spark-defaults.conf and then months later had forgotten about this.

There is a cascading hierarchy whereby SparkConf settings get precedence, then spark-submit settings and then spark-defaults.conf. spark-defaults.conf overrides defaults set by Apache Spark team

Once I removed the settings from spark-defaults, all was fixed.

Answer 2

It is because of the maximum of your physical memory .

your spark memory in spark UI 14.6GB So you must request memory for each executor below the volume 14.6GB, for this you can add config to your spark conf something like this:

conf.set("spark.executor.memory", "10gb")

if you request more than your physical memory spark does't allocate cpu cores to your job and display 0 in Cores in spark UI and run NOTHING.