Spark - Yarn 客户端模式下应用程序主节点的执行程序数量

Question

Question

In Yarn client mode, how many executors are required for the application master (AM)?

In Yarn cluster mode, according to How-to: Tune Your Apache Spark Jobs (Part 2) , the AM requires 2 executors.

A better option would be to use --num-executors 17 --executor-cores 5 --executor-memory 19G. Why?

This config results in three executors on all nodes except for the one with the AM, which will have two executors.

Update

I misunderstood the article. AM is a JVM, so if running 3 JVM (executor) per node, then AM node can run 2 executors. YARN mode has nothing to do with how many non-executor JVM required for AM process. Always 1 JVM.

Answer 1

My Guess is that your execution mode (cluster / client) has nothing to do with the AM in terms of impact. Yarn Application Master always runs on the cluster and is decorrelated from the driver/client (which is the only one impacted by your exec mode)

1 - As you can see below Client and App Master are decorelated

2 - Spark Cluster mode : Driver is executer on a driver node

3 - Spark Client Mode : Driver is executed from the worker/machine where the spark job submission originated from