Spark 内存不足异常

Question

I receive 10 GB file everyday containing employee details.Need to select the latest records from previous day and current day file.Ex: 6th August and 7th August files need to compared on time-Stamp column and select latest records

• 6th August File

  emp-id name dept phone-No time-Stamp 1 Jhon Sales 817234518 12-6-2019 2 Marry Production 927234565 4-3-2019 3 James Marketing 625234522 21-1-2019

• 7th August File

  emp-id name dept phone-No time-Stamp 1 Jhon Sales 817234518 12-7-2019 4 Jerry Sales 653214442 12-7-2019 3 James Marketing 625234522 2-6-2019

• Expected output

  emp-id name dept phone-No time-Stamp 1 Jhon Sales 817234518 12-7-2019 2 Marry Production 927234565 4-3-2019 3 James Marketing 625234522 2-5-2019 4 Jerry Sales 653214442 12-7-2019

I tried solutions as below and got expected result.

 val mergedDF = currentDayDF.union(previousDayDF)

mergedDF.show(false)

val windowSpec = Window.partitionBy("emp-id").orderBy(col("timeStamp").desc)
val latestForEachKey = mergedDF.withColumn("rank", rank().over(windowSpec))
                               .filter(col("rank") === 1)
                               .drop("rank")

Questions

1. Every day's input file size is 10 GB, What if cluster memory(total executors memory) is less than 20 GB to load both data sets (previous day and current day) will spark throw Out of Memory exception?

   I thought, spark divides large file as partitions to process, hence at the beginning only few partitions are loaded into executors memory, transformations applied and saved intermediate resultant dataset to secondary memory and then process is continued for remaining partitions.But parition by requires all the partitions of data as its wider transformation my guess is wrong. So will spark throw OOM exception?

Answer 1

Partitions are used for parallel execution. Spark will try to load all the 20GB data simultaneously in all the available partitions. If the combined memory of all executors where partitions are created is less than 20 GB then it will throw out of memory error