调整 Spark：当可用内核不均匀时每个节点的执行器数量

Question

I have read that having 5 cores per Executor in Spark achieves the optimal read/write throughput - so setting spark.executor.cores = 5 is usually desired. And also that you should subtract one core per node to allow for the underlying daemon processes to run.

So, determining the number of executors per node follows this formula:

executors per node = (cores per node - 1) / 5 cores per executor

However, what is the best approach in a scenario where you have 8 cores in each node machine?

1.4 executors per node = (8 - 1) / 5

First question - will Spark/yarn have an executor spanning multiple nodes?

If not - then I need to round. Which way should I go? It seems my options are:

1.) round down to 1 - meaning I'd only have 1 executor per node. I could increase the cores per executor, though don't know if I would get any benefit to that.

2.) round up to 2 - that means I'd have to decrease the cores per executor to 3 (8 cores available, - 1 for the daemons, and can't have 1/2 a core), which could decrease their efficiency.

Answer 1

Here spark.executor.cores = 5 is not a hard-lined value. Thumb rule is # of cores equal to or less than 5 .

We need 1 core for OS & other Hadoop daemons. We are left with 7 cores per node. Remember we need 1 executor for YARN out of all the executors.

When spark.executor.cores = 4 we cannot leave 1 executor for YARN, so I suggest I not take up this value.

When spark.executor.cores = 3 or spark.executor.cores = 2 after leaving one node for YARN we will always be left out with 1 executor per node.

Now which one is efficient for your code. Well that cannot be interpreted, it depends on multiple other factors like the amount of data used, # of joins used etc.

This is based on my understanding. It provides a start to explore multiple other options.

NOTE: If you are using some outer Java libraries & Datasets in your code, you might need to have 1 core per executor for preserving the type safety.

Hope it helps...