How to change the number of CPUs each worker uses in Spark Standalone mode?

Question

I could run master and slave nodes in Spark Standalone mode on the machine with OS Windows 7, 8 cores CPU and 16 GB RAM.

Now I'm trying to change the number of workers on this machine.

As I read on Spark's site, launching scripts are not currently supported in Windows OS. So I start the master and workers manually.

In driver program I define:

SparkConf conf = new SparkConf().setAppName("myapplication")
  .setMaster("spark://fujitsu11:7077")
  .setJars(new String[] {"target/maven1-0.0.1-SNAPSHOT-driver.jar"})
  .set("spark.home","D:/spark")
  .set("spark.executor.memory", "2g")
  .set("spark.worker.instances","2")
  .set("spark.worker.cores", "4"); // I allocate four cores for each worker
JavaSparkContext sc = new JavaSparkContext(conf);

Then in Cygwin I start by hand master daemon and two workers.

But when I run my application and follow on http://localhost:8080 I see, that both workers trying to use all 8 cores...

How could I change the number of cores the instances use?

Answer 1

The number of the CPU/cores a worker uses is given as an environment variable at start time, and you could set it in two places:

1. In conf/spark-env.sh as SPARK_WORKER_CORES .
2. As an command line argument when you start a worker daemon as --cores 4 .