在 Java Spark 中收集数据集时出现 OutOfMemoryError GC 开销限制

Question

I have some data of approximate size 250MB.

I want to load the data and convert it to a map

class MyData implements Serializable {

    private Map<String, List<SomeObject>> myMap;

    MyData(SparkSession sparkSession, String inputPath) {

        Dataset<Klass> ds = sparkSession.read().json(inputPath).as(Encoders.bean(Klass.class));
        myMap = ds.collectAsList().stream().collect(Collectors.toMap(
                                    Klass::getField1(),
                                    Klass::getField2()
                            )
                    );
    }
}

This is my spark execution configuration

--master yarn --deploy-mode cluster --executor-cores 2 --num-executors 200 --executor-memory 10240M

Is it not a good practice to convert dataset to a list/map? Or is it a configuration issue? Or a code issue?

Answer 1

It looks like your collecting all the data in the Dataset into the Spark driver with:

myMap = ds.collectAsList()...

Therefore you should set the driver memory with --driver-memory 2G on the command line (aka your "spark execution configuration".

The default value for this parameter is 1G which is likely not quite enough for 250M of raw data.