Spark 资源分配：内核数

Question

Require understanding on how to configure Cores for an Spark Job . My Machine can have a max. of 11 Cores, 28 Gb memory . Below is how I'm allocating resources for my Spark Job and it's execution time is 4.9 mins

--driver-memory 2g \
--executor-memory 24g \
--executor-cores 10 \
--num-executors 6

But I ran through multiple articles mentioning number of cores should be ~ 5, when I ran job with this configuration it's execution time increased to 6.9 mins

--driver-memory 2g \
--executor-memory 24g \
--executor-cores 5 \
--num-executors 6 \

1. Will there be any issue keeping Number of Cores close to Max. value (10 in my case)?
2. Are there any benefits of keeping No. of Cores to 5, as suggested in many articles?
3. So in general what are the factors to consider in determining Number of cores?

Answer 1

It all depends on the behaviour of job, one config does not optimise all needs.

--executor-cores means no of cores on 1 machine.

It that number is too big (>5) then the machine's disk and.network (which will be shared among all executor spark cores on that machine) will create bottleneck. If that no is too less (~1) then it will not achieve good data parallelism and won't benefit from locality of data on same machine.

TLDR : --executor-coers 5 is fine.