将大文件拆分为小文件，并使用spark保存在不同的路径中

Question

How to split a large file/RDD/DF into small files and save to different paths.

ex: If there is a file that contains usernames(single column)in a text file and wants to split that into N files and write that N files into different directories.

val x=20
val namesRDD=sc.textfile("readPath")
val N = namesRDD.count/x

How to split the namesRDD into N files and write those to some "savepath/N/" - ie first file is written to "savepath/1/", the second file is written to "savepath/2/" and so on.

Answer 1

Using repartitionByRange will let you split your data this way.

example:

df.repartitionByRange($"region").write.csv("data/regions")

This will create 1 part file for every region that appears in your data. If you have 10 regions, you will have 10 different part- files.

If you want to specify your own name, you will have to apply your own function to save the file with foreachPartition .

df.repartitionByRange($"region")
  .foreachPartition(region => {
     // custom implementation
  })

Answer 2

split the file/df into N parts using repartition(if there are no columns to do repartitionByRange and want to split randomly)

df.repartition(N)
  .write.text(storePath)

then read those partitions (do whatever on that partitioned Df)

  for (i <- 0 until N) {
    val parts = f"${i}%04d"
    val splitPath = s"${path}/part-0${partNumber}-*"
    //read data from the `splitPath`
  }