在Apache Spark Java中无法.collect（），OutOfMemoryError：超出了GC开销限制

Question

I'm completely new to Spark (running on MacOS) and I've been trying to test out some .parquet files I have in my PC which are around 120mb in size, trying to run them locally with master set to local[*] .

Basically the pipeline of operations I currently have is the following one...

        dataset
            .where(...)
            .toJavaRDD()
            .sortBy(...) // Tried .collect() here. Current 'debug' point.
            .groupBy(...) // Tried .collect() here.
            .flatMapValues(...)
            .mapToPair(...) // Tried .collect() here.
            .reduceByKey(...); // Tried .collect() here.

The first thing I would like to ask, how can I check the parquet file schema? I've read out something that it's possible with Hive but haven't found anything. If you have any resources that can be useful to me it would be really appreaciated.

Secondly, as I don't really know all the parquet column names I must access in groupBy() and such, I'm just trying to collect everything in the first sortBy(), see what comes out and such (some minor testing in order to get started with Spark and how everything works). But as the question says I always get the given error. Is there anything I'm doing wrong? Shouldn't I .collect() after all?

I tried to print at some points but it seems that it goes to some logs as far as I've read and I don't really know if they are stored locally in the computer or how can I access them so I can see the output?

When it comes to the Spark configuration, it's the most basic one.

    final SparkConf sparkConf = new SparkConf()
        .setMaster("local[*]")
        .setAppName("test_app_name");

    final JavaSparkContext jsc = new JavaSparkContext(sparkConf);

    // Get the singleton instance of SparkSession
    SparkSession sparkSession = SparkSession.builder()
            .config(jsc.getConf())
            .getOrCreate();

    Dataset<Row> dataset = sparkSession.read()
            .parquet(inputPath);

Answer 1

Doing a collect action in spark is not good if your data is huge. Collect essentially brings all the data to the driver. This will cause memory outOf bound in the driver. Unless you are absolutely sure that the data you are collecting is pretty small.

To see the dataset/dataframe schema you need to do

 dataset.printSchema

To print a few rows of the dataset you can use the followings

 dataset.show(10) // number of rows you want to see

Or

 dataset.take(10).foreach(println) // takes 10 rows and print

If you want to see random rows for sampling you might use

 df.select("name").sample(.2, true).show(10)

Or

 df.select("name").sample(.2, true).take(10).foreach(println)