面对 spark 上小数据集的大数据溢出

Question

I am trying to run some spark sql on NOA datasets available here:

https://www.ncei.noaa.gov/data/global-summary-of-the-day/access/2021/

I am trying to run some query which involves grouping and sorting.

df
      .groupBy("COUNTRY_FULL")
      .agg(max("rank"), last("consecutive").as("consecutive"))
      .withColumn("maxDays", maxDaysTornodoUdf(col("consecutive")))
      .sort(col("maxDays").desc)
      .limit(1)
      .show()

The input size is just 50 MB zipped csvs and I am running this locally (4 cores). These are the settings I use.

• spark.driver.memory: 14g
• spark.sql.windowExec.buffer.in.memory.threshold: 20000
• spark.sql.windowExec.buffer.spill.threshold: 20000
• spark.sql.shuffle.partitions: 400

I see too many disk spills for such a small data

21/08/16 10:23:13 INFO UnsafeExternalSorter: Thread 54 spilling sort data of 416.0 MB to disk (371  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 79 spilling sort data of 416.0 MB to disk (130  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 53 spilling sort data of 400.0 MB to disk (240  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 69 spilling sort data of 400.0 MB to disk (24  times so far)
21/08/16 10:23:16 INFO UnsafeExternalSorter: Thread 54 spilling sort data of 416.0 MB to disk (372  times so far)
21/08/16 10:23:16 INFO UnsafeExternalSorter: Thread 79 spilling sort data of 416.0 MB to disk (131  times so far)

However, when I check the Spark UI, the spillage doesnt seem to be much

Eventually the spark job terminates with error Not Enough memory I do not understand what is happening.

Answer 1

You are using 400 as spark.sql.shuffle.partitions , which is too much for the data size which you are dealing with.

Having more shuffle partitions for lesser amount of data causes more partitions/tasks and it will reduce the performance. Read best practices to configure shuffle partition here .

Try reducing shuffle partitions. You may try setting it to spark.sparkContext.defaultParallelism .