纱线上火花的作业调度较慢

Question

I have two mapr clusters with the following configuration,

cluster 1: hosted on aws, 3 nodes with 32g of memory/32 cores each
cluster 2: hosted on bare-metal servers, 8 nodes with 128g of memory/32 cores each

I'm running following piece of pyspark code thru yarn, on both the clusters

df=hc.sql("select * from hive_table")
df.registerTempTable("df")
df.cache().count()
for: 100times
    result=hc.sql('select xxxx from df')
    result.write.saveAsTable('some hive table', mode='append')

the above code submits 100 new jobs in spark (running on top of yarn). On cluster one, the whole operation completes in 30 mins, but on cluster 2 which is a bigger one, it takes 90 mins to complete the same operation. Upon checking, I found out that though each job takes almost about the same time (little faster in the cluster 2), the time between each job is way too higher in 2 than 1.

Possible reasons,

1. latency between drivers and executors node? -- I'm running in
   yarn-client mode
2. low driver memory or the way I initiate a spark cluster on yarn is wrong?

how do I submit the jobs?

Cluster 1: /opt/mapr/spark/spark-1.6.1/bin/spark-submit --master yarn --deploy-mode client --num-executors 10 --executor-memory 10g --executor-cores 5 --driver-memory 10g --driver-cores 10 --conf spark.driver.maxResultSize="0" --conf spark.default.parallelism="100" --queue default

Cluster 2: /opt/mapr/spark/spark-1.6.1/bin/spark-submit --master yarn --deploy-mode client --num-executors 10 --executor-memory 80g --executor-cores 28 --driver-memory 25g --driver-cores 25 --conf spark.driver.maxResultSize="0" --conf spark.default.parallelism="100" --queue default

PS: only pasted part of the code. There are other modules in the code. Overall, cluster 2 process the code 3x faster than cluster 1, so I don't the think the issue with the 'general' speed.

my question is more specific to 'time' between the jobs. For ex, above piece of code runs 100 spark-sql jobs, each job takes on an average 2s in cluster 2, 5s in cluster 1. Time between each job is way too higher in cluster 2 compared to cluster 1.

Answer 1

In your pseudo-code I don't see any driver related actions (assuming that the executors save the data to a distributed FS)

Please note that:

1. You df.cache() but it doesn't seems like you are using the cached df.
2. Your yarn-client configuration seems to be problematic.

It looks like you are trying to use more executor memory and cores than available.

In cluster #1, there are 3 nodes with 32GB of RAM, your execution code is: --num-executors 10 --executor-memory 10g

Best case scenario you'll have 9 executors with 10GB of RAM each. Max 3 executors on each node. I'd assume that you'll be able to execute only 2 executor per node (as from the 32GB of RAM more than 2GB will be used for yarn, overhead, etc, hence less than 29GB will be left ==> 2 containers of 10GB each)

==> Cluster #1 will have 6 to 9 executors

In cluster #2, there are 5 nodes with 128GB of RAM, your execution code is: --num-executors 10 --executor-memory 80g

Best case scenario you'll have 5 executors with 80GB of RAM. Each executor on one node.

Since cluster #1 has more executors (even that they are smaller) , it is possible that it will run faster (depend on your application)

Reducing executors memory and core in cluster #2 together with increasing in the number of executors should provide better performance.