在纱线上使用火花时火花执行器和纱线容器是什么关系

Question

what is the relationship between spark executor and yarn container when using spark on yarn?
For example, when I set executor-memory = 20G and yarn container memory = 10G, does 1 executor contains 2 containers?

Answer 1

Spark Executor Runs within a Yarn Container. A Yarn Container is provided by Resource Manager on demand. A Yarn container can have 1 or more Spark Executors. Spark-Executors are the one which runs the Tasks. Spark Executor will be started on a Worker Node(DataNode)

In your case when you set executor-memory = 20G -> This means you are asking for a Container of size 20GB in which your Executors will be running. Now you might have 1 or more Executors using this 20GB of Memory and this is Per Worker Node .

So for example if u have a Cluster to 8 nodes, it will be 8 * 20 GB of Total Memory for your Job.

Below are the 3 config options available in yarn-site.xml with which you can play around and see the differences.

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb
yarn.nodemanager.resource.memory-mb

Answer 2

When running Spark on YARN, each Spark executor runs as a YARN container, This means the number of containers will always be the same as the executors created by a Spark application eg via --num-executors parameter in spark-submit.

https://stackoverflow.com/a/38348175/9605741

Answer 3

In YARN mode, each executor runs in one container. The number of executors is the same as the number of containers allocated from YARN(except in cluster mode, which will allocate another container to run the driver).