Where to set “spark.yarn.executor.memoryOverhead”

Question

I am getting following error while running my spark-scala program.

YarnSchedulerBackends$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 2.6GB of 2.5GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

I have set spark.yarn.executor.memoryOverhead in the program while creating SparkSession.

My question is - is it ok to set "spark.yarn.executor.memoryOverhead" while creating SparkSession or should it be passed during runtime with spark-submit?

Answer 1

You have to set the spark.yarn.executor.memoryOverhead at the time of sparkSession creation. This parameter is used as amount of off-heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. This tends to grow with the executor size (typically 6-10%).

Now this allocation can only be done at the time of allocation of the executor not at the runtime.