Cluster Resource Usage in Databricks

Question

I was just wondering if anyone could explain if all compute resources in a Databricks cluster are shared or if the resources are tied to each worker. For example, if two users were connected to a cluster made up of 2 workers with 4 cores per worker and one user's job required 2 cores and the other's required 6 cores, would they be able to share the 8 total cores or would the full 4 cores from one worker be unavailable during the job that only required 2 cores?

Answer 1

TL;DR; Yes, default behavior is to allow sharing but you're going to have to tightly control the default parallelism with such a small cluster.

Take a look at Job Scheduling for Apache Spark. I'm assuming you are using an "all-purpose" / "interactive" cluster where users are working on notebooks OR you are submitting jobs to an existing, all-purpose cluster and it is NOT a job cluster with multiple spark applications being deployed.

Databricks Runs in FAIR Scheduling Mode by Default

Under fair sharing, Spark assigns tasks between jobs in a “round robin” fashion, so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can start receiving resources right away and still get good response times, without waiting for the long job to finish. This mode is best for multi-user settings.

• By default, all queries started in a notebook run in the same fair scheduling pool
• The Apache Spark scheduler in Azure Databricks automatically preempts tasks to enforce fair sharing.

Apache Spark Defaults to FIFO

By default, Spark's scheduler runs jobs in FIFO fashion. Each job is divided into “stages” (eg map and reduce phases), and the first job gets priority on all available resources while its stages have tasks to launch, then the second job gets priority, etc. If the jobs at the head of the queue don't need to use the whole cluster, later jobs can start to run right away, but if the jobs at the head of the queue are large, then later jobs may be delayed significantly.

Keep in mind the word "job" is specific Spark term that represents an action being taken that launches one or more stages and tasks. See What is the concept of application, job, stage and task in spark? .

So in your example you have...

• 2 Workers with 4 cores each == 8 cores == 8 tasks can be handled in parallel
• One application (App A) that has a job that launches a stage with only 2 tasks.
• One application (App B) that has a job that launches a stage with 6 tasks.

In this case, YES, you will be able to share the resources of the cluster . However, the devil is in the default behaviors. If you're reading from many files, performing a join, aggregating, etc, you're going to run into the fact that Spark is going to partition your data into chunks that can be acted on in parallel (see configuration like spark.default.parallelism ).

So, in a more realistic example, you're going to have...

• 2 Workers with 4 cores each == 8 cores == 8 tasks can be handled in parallel
• One application (App A) that has a job that launches a stage with 200 tasks .
• One application (App B) that has a job that launches three stage with 8, 200, and 1 tasks respectively .

In a scenario like this FIFO scheduling, as is the default, will result in one of these applications blocking the other since the number of executors is completely overwhelmed by the number of tasks in just one stage.

In a FAIR scheduling mode, there will still be some blocking since the number of executors is small but some work will be done on each job since FAIR scheduling does a round-robin at the task level.

In Apache Spark, you have tighter control by creating different pools of the resources and submitting apps only to those pools where they have "isolated" resources. The "better" way of doing this is with Databricks Job clusters that have isolated compute dedicated to the application being ran.