Why spark environment parameters is not consistent with executors?

Question

I'm running my spark application on EMR. In spark history UI, under the environment tab, spark.executor.instances equals 1 . However, under the executors tab, it shows that there were total 9 executors , including 1 driver.

Why does this happen?

Answer 1

spark.executor.instances this is set to the initial number of core nodes plus the number of task nodes in the cluster.

executors is the number of tasks that each executor can execute in parallel.

This link will help explain the various meanings and options:

Submitting User Applications with spark-submit