如何解决 EMR Spark Out Of Memory 错误

Question

I have a Spark job which I am trying to execute on EMR. It is giving me the below error:

java.lang.OutOfMemoryError: Java heap space
-XX:OnOutOfMemoryError="kill -9 %p"
Executing /bin/sh -c "kill -9 22611"...

I have tried it with even 10 core instances of type m5.12xlarge but still the same issue. My code is working fine as I have tested it via AWS Glue and that has succeeded with G1.X and 20 DPUs (takes around 3 hours to complete the job). Any recommendation regarding how I choose the EMR instance type?

Answer 1

So changing the instance types only won't always help, we need to play with spark configurations as well. I followed something like mentioned here and the job is successful on EMR.