EMR 上 Spark 批处理作业的优化

Question

We are running a spark-job on the EMR cluster with the Cluster configuration as given below.

Resources:
Node Type:CORE - 2 INSTANCES OF
r4.8xlarge
32 vCore, 244 GiB memory, EBS only storage
EBS Storage:32 GiB

Node Type: MASTER
1 Instance of r4.4xlarge
16 vCore, 122 GiB memory, EBS only storage
EBS Storage:32 GiB

Node Type: TASK- 
2 INSTANCES Of 
r4.4xlarge
16 vCore, 122 GiB memory, EBS only storage
EBS Storage:32 GiB

We are doing spark-submit by using the following arguments on the EMR Console:

/usr/bin/spark-submit --deploy-mode cluster --conf spark.sql.parquet.fs.optimized.committer.optimization-enabled=true --conf spark.sql.files.ignoreCorruptFiles=true --driver-memory 5g --master yarn --class class_name s3://location_of_jar -c s3://location of input to jar -w xyz.json

We feel these arguments are not leveraging the use of available full available resources. Can any one please suggest is there any other optimized way to do spark-submit on EMR by changing any of the spark-defaults.conf file or by passing more arguments so that there is optimum utilization of all available resources? We run one job at once. There are no parallel jobs running on the cluster

Answer 1

Without knowing the resources allocated per executor, the nature of the job, the volume of data you're processing etc., it's very hard to give a proper suggestion. I think the best you can do now, is to also install ganglia while creating a EMR cluster. The ganglia web ui is available via http://master-public-dns-name/ganglia/

Look at the cpu and memory usage, to start with. That will give you a good enough idea if you're optimally allocating resources for your spark job, and then tune resources per executor accordingly.

The number of executors, executor memory and cores can be set in your spark-submit command using the following way (these are sample values):

--num-executors 10
--executor-cores 1
--executor-memory 5g

After looking at the ganglia charts you'll get a sense which resource is being under/over-utilized. Change those accordingly.

If you don't want to play around with these numbers and let spark decide what's the best combination, it might be worth setting dynamic resource allocation to true using the following lines:

--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=true

One thing to note here is that yarn will get 75% of the total memory allocated to core + task nodes. Also, the driver and each executor has a memory overhead associated to it. Look up the spark documentation for that. Keep that in mind while manually allocating resources to driver and executor(s).

Answer 2

The first step to analyse spark job is spark-ui . so use the tracking url and look into the l ogs, jobs, executors, streaming .

http://cluster_manager_host:8088/

For more detailed analysis of memory and cpu utilisation, You can use Gangalia tool too.

http://cluster_manager_host/Gangalia

After this what you can do is:

• You have to go for custom configurations like

  (i) No of executors --conf num-executors x

  (ii) Executors memory --conf executor-memory y

  (iii) No of cores -- conf executor-cores z

  (iv) Enable dynamic resource allocation --conf spark.dynamicAllocation.enabled=true

  (v) Enable max resource allocation --conf maximizeResourceAllocation=true

  (vi) Change the serialisation from default to Kryo , --conf org.apache.spark.serializer.KryoSerializer

  (vii) Change the num of partition from default to custom based on your configuration rdd=rdd.repatition(sparkConf.defaultParalallism*2)

• If after proper configuration of above your job is still slow then, Please Change your code and use proper functions and objects. like

  (i) If you are sending data to any external destination like Kinesis, DB or Kafka, Use mapPartitions or foreachPatitions and reduce no of object creations.

  (ii) If you are calling external API then also follow the above strategy.

  (iii) Use of proper Data structures.

For more Information, You can refer: here

I hope this will help you.