yarn - spark parallel job

Question

I made yarn-cluster which has only 1 work node, and it seems to work fine when I submit my spark application job. When I submit job more than one, jobs are on hadoop queue and process submitted application one by one. I want to process my applications parallelly, not one by one. Is there any configuration for this? or unable to do this on yarn?

Answer 1

Yarn submits jobs one by one by default. For submit multiple jobs you can change amount of your executor cores:

spark-submit class /jar --executor-memory 2g --num-executors 15 --executor-cores 3 --master yarn --deploy-mode cluster 

You also can change this properties in your yarn-site.xml