Hadoop YARN slow resource allocation for Spark

Question

I have this problem where YARN is very slow in allocating resource for spark, sometimes it's late to add executor. This started after a black out, after I restart the cluster, it started acting slow even though I restart it a few times. No configuration were changes. It was working fine before.

The picture above shows executor were added very late. Sometimes some of the executor were not added at all. The picture below taken before the black out.

Can anyone figure out what's happening? it's still working, but much-much slower than before.

Answer 1

Yarn has Dynamic resource allocation property. I am not sure whether you have enabled this property or not. If enabled means yarn will allocate the resources based on the load.