spark的spark调度模式和应用程序队列之间有什么区别？

Question

While testing the behavior of spark jobs when multiple jobs are submitted to run concurrently or smaller jobs submitted later. i came across two settings in spark ui. one is scheduling mode available withing spark as shown in below image

And one is under scheduler as show below

I want to understand the difference between two settings and preemption. My Requirement is that while running the bigger job, small jobs submitted in between must get the resources without waiting longer.

Answer 1

Let me explain it for the Spark On Yarn mode.

When you submit a scala code to spark, spark client will interact with yarn and launch a yarn application. This application will be duty on all the jobs in your scala code. In most cases, each job correspond to an Spark Action like reduce()、collect().Then ,the problem comes, how to schedule different jobs in this application, for example, in your application , there a 3 concurrent jobs comes out and waiting for execution? To deal with it , Spark make the scheduler rule for job, including FIFO and Fair.That is to say , spark scheduler ,including FIFO and Fair, is on the level of job , and it is the spark ApplicationMaster which is do the scheduling work.

But yarn's scheduler, is on the level of Container .Yarn doesn't care what is running in this container, maybe the container it is a Mapper task , a Reducer task , a Spark Driver process or a Spark executor process and so on. For example, your MapReduce job is currently asking for 10 container, each container need (10g memory and 2 vcores), and your spark application is currently asking for 4 container ,each container need (10g memory and 2 vcores). Yarn has to decide how many container are now available in the cluster and how much resouce should be allocated for each request by a rule ,this rule is yarn's scheduler, including FairScheduler and CapacityScheduler.

In general, your spark application ask for several container from yarn, yarn will decide how many container can be allocated for your spark application currently by its scheduler.After these container are allocated , Spark ApplicationMaster will decide how to distribute these container among its jobs.

Below is the official document about spark scheduler: https://spark.apache.org/docs/2.0.0-preview/job-scheduling.html#scheduling-within-an-application

Answer 2

I think Spark.scheduling.mode (Fair/FIFO), shown in the figure, is for scheduling tasksets (single-same stage tasks) submitted to the taskscheduler using a FAIR or FIFO policy etc.. These tasksets belong to the same job.

To be able to run jobs concurrently, execute each job (transformations + action) in a separate thread. When a job is submitted to the DAG the main thread is blocked until job completes and result is returned or saved.