What is the theoretical max parallelism of tasks in my Foundry Job?

Question

I know there's indications of parallelism (Task Concurrency) in my job's Spark Details page, but I'm wondering how this number is calculated since it doesn't match the number of Executors my job is running with?

Answer 1

There are 3 things that influence this:

1. TASK_CPUs (T)

• Task CPUs control the number of cores given to an individual task. A typical setup will have 1 for this setting, meaning each task will operate with a single core.

2. EXECUTOR_CORES (C)

• The number of cores allocated to each Executor running your job. A typical setup will have 2 cores per Executor for your job

3. NUM_EXECUTORS (E)

• The number of executors allocated to your job. A typical setup will have 2 Executors for your job.

These are used together in your Foundry job like so:

• The total number of cores available to do work in your cluster is C * E , therefore is typically 4 (2 * 2).
• The amount of parallelism is the total number of cores / cores per task, or (C * E) / T , therefore typically (2 * 2) / 1 , or 4

You therefore will typically see a max parallelism of 4 in your jobs; increasing your Executor count will therefore boost your max parallelism. Be wary of boosting your cores per executor, you may encounter problems