Spark失败了java.lang.OutOfMemoryError：超出了GC开销限制？

Question

This is my java code in which I am querying data from Hive using Apache spark sql.

JavaSparkContext ctx = new JavaSparkContext(new SparkConf().setAppName("LoadData").setMaster("MasterUrl"));
HiveContext sqlContext = new HiveContext(ctx.sc());
List<Row> result = sqlContext.sql("Select * from Tablename").collectAsList();

when I run this code it throws java.lang.OutOfMemoryError: GC overhead limit exceeded. How to solve this or how to increase the memory in Spark configuration.

Answer 1

If you are using the spark-shell to run it then you can use the driver-memory to bump the memory limit:

spark-shell --driver-memory Xg [other options]

If the executors are having problems then you can adjust their memory limits with --executor-memory XG

You can find more info how to exactly set them in the guides: submission for executor memory, configuration for driver memory.

@Edit: since you are running it from Netbeans you should be able to pass them as JVM arguments -Dspark.driver.memory=XG and -Dspark.executor.memory=XG . I think it was in Project Properties under Run .

Answer 2

have you found any solutions for your issue yet? please share them if you have :D

and here is my idea: rdd and also javaRDD has a method toLocalIterator() , spark document said that

The iterator will consume as much memory as the largest partition in this RDD.

it means iterator will consume less memory than List if the rdd is devided into many partitions, you can try like this:

Iterator<Row> iter = sqlContext.sql("Select * from Tablename").javaRDD().toLocalIterator();
while (iter.hasNext()){
    Row row = iter.next();
    //your code here
}

ps: it's just an idea and i haven't tested it yet