Yarn 是否根据我们在 spark-submit 命令中传递的 executor 数量为 application master 分配一个容器

Question

Lets assume that I am submitting a Spark application in yarn-client mode. In Spark submit I am passing the --num-executors as 10. When the client submits this spark application to resourceManager,

Does resource manager allocate one executor container for Application master process from the --num-executors(10) and teh rest 9 will be given for actual executors?

or

Does it allocate one new container for application master or give 10 containers for executors alone?

Answer 1

--num-executors is to request that number of executors from a cluster manager (that may also be Hadoop YARN). That's Spark's requirement.

An application master (of a YARN application) is just a thing of YARN.

It may happen that a Spark application can also be a YARN application. In such case, the Spark application gets 10 containers and one extra container for the AM.