Spark Local与Cluster

Question

What is the Spark cluster equivalent of standalone's local[N]. I mean, the value we set as a parameter of local as N, which parameter takes it in the cluster mode?

Answer 1

In local[N] - N is the maximum number of cores can be used in a node at any point of time.

In cluster mode you can set --executor-cores N . It means that each executor can run a maximum of N tasks at the same time in an executor.

Answer 2

In cluster mode, one executor will run on one worker node, which means that one executor will takes all the cores on the worker node. It could result in under-utilization of the resources. Keep in mind, driver will also takes one worker node.