How does spark.dynamicAllocation.enabled influence the order of jobs?

Question

Need an understanding on when to use spark.dynamicAllocation.enabled - What are advantages and disadvantages of using it? I have queue where jobs get submitted.

1. 9:30 AM --> Job A gets submitted with dynamicAllocation enabled.
2. 10:30 AM --> Job B gets submitted with dynamicAllocation enabled.

Note: My Data is huge (processing will be done on 10GB data with transformations).

Which Job gets the preference on allocation of executors to Job A or Job B and how does the spark co-ordinates b/w 2 applications?

Answer 1

Dynamic Allocation of Executors is about resizing your pool of executors.

Quoting Dynamic Allocation :

spark.dynamicAllocation.enabled Whether to use dynamic resource allocation, which scales the number of executors registered with this application up and down based on the workload.

And later on in Dynamic Resource Allocation :

Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application may give resources back to the cluster if they are no longer used and request them again later when there is demand. This feature is particularly useful if multiple applications share resources in your Spark cluster.

In other words, job A will usually finish before job B will be executed. Spark jobs are usually executed sequentially, ie a job has to finish before another can start.

Usually...

SparkContext is thread-safe and can handle jobs from a Spark application. That means that you may submit jobs at the same time or one after another and in some configuration expect that these two jobs will run in parallel.

Quoting Scheduling Within an Application :

Inside a given Spark application (SparkContext instance), multiple parallel jobs can run simultaneously if they were submitted from separate threads. By “job”, in this section, we mean a Spark action (eg save, collect) and any tasks that need to run to evaluate that action. Spark's scheduler is fully thread-safe and supports this use case to enable applications that serve multiple requests (eg queries for multiple users).

By default, Spark's scheduler runs jobs in FIFO fashion. Each job is divided into “stages” (eg map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc.

it is also possible to configure fair sharing between jobs. Under fair sharing, Spark assigns tasks between jobs in a “round robin” fashion, so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can start receiving resources right away and still get good response times, without waiting for the long job to finish. This mode is best for multi-user settings.

---

Wrapping up...

Which Job gets the preference on allocation of executors to Job A or Job B and how does the spark co-ordinates b/w 2 applications?

Job A.

Unless you have enabled Fair Scheduler Pools :

The fair scheduler also supports grouping jobs into pools, and setting different scheduling options (eg weight) for each pool. This can be useful to create a “high-priority” pool for more important jobs, for example, or to group the jobs of each user together and give users equal shares regardless of how many concurrent jobs they have instead of giving jobs equal shares.