SPARK SQL java.lang.OutOfMemoryError：超出GC开销限制

Question

I try to fetch the data for one day to test and create a temp table using Spark SQL. After that, I am performing my queries on temp table. So that process can be fast but still getting GC memory issue.

Spark configuration:

sparkConf = new SparkConf().setMaster("local[1]").setAppName("Scheduler"); 
sparkConf.set("spark.driver.memory", "2g") ; 
sparkConf.set("spark.executor.memory", "1g"); 
sparkConf.set("spark.memory.onHeap.size","1g") ; 
sparkConf.set("spark.num.executors", "4"); 
sparkConf.set("spark.executor.cores", "1"); 
sparkConf.set("spark.serializer", KryoSerializer.class.getName());

Answer 1

OoM can happen also when the GC activity consumes a certain percentage of CPU resource. I would try to increase the heap and/or use a profiling tool jconsole or visualvm to check what is the root cause