Spark EMR：需要配置多个 spark-submits 才能在 EMR 集群中并行工作

Question

I am new to AWS EMR and I face a problem of making multiple spark-submits to run parallel.

I have some jobs scheduled to run every 10 minutes and a job that runs every 6 hours. The cluster has enough resources to run them all at the same time, but the default config puts them into the single root. default queue that makes them run sequentially. This is not what I want. What do I have to write in configuration files?

I've tried adding queues "1", "2" and "3" to root queue (in yarn-site.xml) and spark-submitting each job into a separate queue. But they still run sequentially (not parallel as I want).

spark-submit --queue 1 --num-executors 1  s3://bucket/some-job.py

spark-submit --queue 2 --num-executors 1  s3://bucket/some-job.py

Answer 1

I've found an unexpected solution.

Seems like current Hadoop documentation does not reflect correct configs for yarn in AWS EMR, so I've used trial and error method to find a way to make it work.

Instead of Capacity scheduler I used Fair scheduler. But it still places all apps in same queue ("pool"), so I had to manually schedule each job into separate queue and configure those queues to eat appropriate amount of resources. This is what I've done:

yarn-site.xml

<property>
   <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

<property>
<name>yarn.scheduler.fair.allocation.file</name>
<value>fair-scheduler.xml</value>
</property>

<property>
<name>yarn.scheduler.fair.preemption</name>
<value>true</value>
</property>

The purpose of Fair scheduler - to schedule tasks and distribute resources fairly. But (surprise lmao) it does not do it if there is no preemption enabled. The first task eats all resources and does not give them away until it finishes if you explicitly don't ASK FOR THEM.

That's how I managed preemption:

fair-scheduler.xml

<allocations>
  <pool name="smalltask">
    <schedulingMode>FAIR</schedulingMode>
    <maxRunningApps>4</maxRunningApps>
    <weight>1</weight>
    <fairSharePreemptionThreshold>0.4</fairSharePreemptionThreshold>
    <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
  </pool>

  <pool name="bigtask">
    <schedulingMode>FAIR</schedulingMode>
    <maxRunningApps>2</maxRunningApps>
    <fairSharePreemptionThreshold>0.6</fairSharePreemptionThreshold>
    <fairSharePreemptionTimeout>1</fairSharePreemptionTimeout>
    <weight>2</weight>
  </pool>
</allocations>

Now I have 2 queues (big and small), small queue can run 4 small tasks, and big queue can run 2 big tasks at the same time. Big queue weights more, so it demands more resources. If small queue takes more than 40% of resources, other queues begin "nationalizing" it and take resources away. Same for big queue (60%). What happens inside each queue I don't know for sure, but seems like resources try to be distributed equally between apps.

My happy new year wish would be detailed documentation of hadoop and EMR.

Answer 2

I have done both but none has worked for me.

Closest is CAPACITY SCHEDULER. I have 4 queues, default, gold, silver, bronze. Individual spark submit (in fact , 4 in a series that is my unit) work well with each queue with a --queue gold|silver option .

But when I run 2 set (1 in gold and 1 in silver, say) both of them eventually hangs.

Regards, Suvo

Answer 3

Simply, give the configuration for the queues. When you create the EMR, you can give the configurations for yarn.scheduler. If you have preferred configurations, then specify it. For example ,

"Classification": "capacity-scheduler",
"Properties": {
     "yarn.scheduler.capacity.root.queues": "default, gold, silver, bronze"
}

this will give you several queue chennels.

---

Another option is to modify the EMR where it is already running. Similar to the above, but it can be done by AWS CLI or other SDKs. See the article .

It uses the command

aws emr modify-instance-groups --cli-input-json file://some.json

with some.json as a form such as:

{
   "ClusterId":"j-MyClusterID",
   "InstanceGroups":[
      {
         "InstanceGroupId":"ig-MyMasterId",
         "Configurations":[
            {
               "Classification":"capacity-scheduler",
               "Properties":{
                  "yarn.scheduler.capacity.root.queues":"default, bronze, silver, gold"
               },
               "Configurations":[]
            }
         ]
      }
   ]
}