How to change number of cores in spark?

Question

I have the following code

import com.datastax.spark.connector._
sc.cassandraTable("test", "hello").cassandraCount

This take about an 1hr 30mins to finish but during the process it produces the following output

[Stage 0:> (12 + 2) / 26674]

where 26674 is the total number of tasks and 12 are the completed tasks The most important one in the number 2 which is the number of tasks that can run in parallel other words max number of cores this application can use is 2. so my big question is How to change this ?

I have 1 spark master node and 2 spark worker nodes Each Spark worker node has 4vCPUs and 16GB RAM (m4.xlarge).

I tried messing around with different values for the following flags

--executor-cores --total-executor-cores --num-executors

but I wasn't able to change 2 to something higher Any ideas?

I use Spark Stand alone cluster

Answer 1

conf / spark-env.sh中的export SPARK_WORKER_CORES=12是超额预订可以并行运行的内核数量的方法