Spark作业抛出“ java.lang.OutOfMemoryError：超出了GC开销限制”

Question

I have a Spark job that throws "java.lang.OutOfMemoryError: GC overhead limit exceeded".

The job is trying to process a filesize 4.5G.

I've tried following spark configuration:

--num-executors 6  --executor-memory 6G --executor-cores 6 --driver-memory 3G 

I tried increasing more cores and executor which sometime works, but takes over 20 minutes to process the file.

Could I do something to improve the performance? or stop the Java Heap issue?

Answer 1

Only solution is to fine tune the configuration.

As per my experience I can say the following points for OOM:

• cache an RDD only if you are going to use it more than once

Still if you need to cache then consider then analyze the data and application with respect to resources.

• If your cluster has enough memory then increase the spark.executor.memory to its max
• Increase the no of partitions to increase the parallelism
• Increase the dedicated memory for caching spark.storage.memoryFraction . If lot of shuffle memory is involved then try to avoid or split the allocation carefully
• Spark's caching feature Persist(MEMORY_AND_DISK) is available at the cost of additional processing (serializing, writing and reading back the data). Usually CPU usage will be too high in this case

Answer 2

1. You can try increasing the driver-memory . If you don't have enough memory may be you can reduce it from executor-memory

2. Check the spark-ui to see what is the scheduler delay. You can access the spark UI on port 4040. If the scheduler delay is high, quite often, the driver may be shipping large amount of data to the executors. Which needs to be fixed.