what is best way to calculate --executor-memory --num-executors --executor-cores in spark

Question

I have cluster contain 1 master and 5 slave ( node ) , each of them 32 core and 64 GB memory .

Is there any pattern to calculate following parameter in spark submission with yarn

--executor-memory --num-executors --executor-cores

Answer 1

If we have following hardware then calculate spark

• 6 Node
• 16 code
• 64 GB of Ram

Calculations:

• 5 core per executor
• -For max HDFS throughput
• Cluster has 6*15 =90 cores in total
• afer taking out Hadoop /Yarn daemon cores)
• 90 cores /5 cores/executor (19/5=18-1)
• =18 executors
• Each node has 3 executors
• 63 GB/3 =21 GB ,21*(1 -0,07)
• ~19 GB
• 1 executor for AM=> 17 executor

---

Ans

• 17 Executors in total
• 19 GB memory /executor
• 5 cores /executor

---

• Number of executer (--num -executors)
• Coures for each executer(--executorr-cores)
• Memory for each executer (--executor-memory)

---

Executer-memory controls the heap size:

• Node some overhead(controlledby spark.yarn.executor.memory.overhead) for off heap memory default is max (384 MB ,.07* Spark.executer.memory)

---

Spark Memory Management

• https://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/

• https://help.talend.com/reader/MFikL3ufPzqRQ0FnpvF7IA/UfPZ4ym6etEiI5zdNxcksA

• https://community.hortonworks.com/articles/42803/spark-on-yarn-executor-resource-allocation-optimiz.html