Spark - How many executors for application master in Yarn client mode

Question

Question

In Yarn client mode, how many executors are required for the application master (AM)?

In Yarn cluster mode, according to How-to: Tune Your Apache Spark Jobs (Part 2) , the AM requires 2 executors.

A better option would be to use --num-executors 17 --executor-cores 5 --executor-memory 19G. Why?

This config results in three executors on all nodes except for the one with the AM, which will have two executors.

Update

I misunderstood the article. AM is a JVM, so if running 3 JVM (executor) per node, then AM node can run 2 executors. YARN mode has nothing to do with how many non-executor JVM required for AM process. Always 1 JVM.

Answer 1

My Guess is that your execution mode (cluster / client) has nothing to do with the AM in terms of impact. Yarn Application Master always runs on the cluster and is decorrelated from the driver/client (which is the only one impacted by your exec mode)

1 - As you can see below Client and App Master are decorelated

2 - Spark Cluster mode : Driver is executer on a driver node

3 - Spark Client Mode : Driver is executed from the worker/machine where the spark job submission originated from