What is the difference between spark.executor.cores and executor-cores in the spark-submit command?

Question

Is there any difference between executor-cores and spark.executor.cores used in spark-submit command?

My job is failed because of GC overhead memory error, so i am trying to increase cores and memory settings.

The total volume i am processing is 50 M records in two files.

Answer 1

The executor-cores flag used in the spark-submit command simply set the spark.executor.cores setting on the Spark-configuration. So they have the same effect :)

A couple of things you may try:

1) You have tagged the question with YARN, so if you find that you are not utilizing all of your cores, you should have a look at Apache Hadoop Yarn - Underutilization of cores

2) Many memory issues on YARN are solved when you increase the memory-overhead by explicitly setting spark.yarn.executor.memoryOverhead. It will default to max(386MB, 0.10* executorMemory) and that is frequently not enough.