Scala - 如何合并 HDFS 位置的增量文件

Question

My requirement is that I've a multiple HDFS location which ingest files from Kafka every hour. So for each directory how to merge all the files of a particular timestamp to current timestamp as a single parquet file and next time merge the files from last merged timestamp to current timestamp only, and repeat the same in future. This all I have to do in Spark Scala job so can't go with normal shell script. Any suggestions are appreciated.

Answer 1

Here is a code snippet that would help to get the thing done.

First step is to the get the list of files per date as a Map. (Map[String, List[String]]) where key is Date and value is list of files with a same date. Date is taken from the modification timestamp of the HDFS file.

Note: Tested the code using local path, give the right HDFS path / url as required.

While writing the output, there is no direct option to specify the target filename, but you can specify the target directory specific to each date. Code makes us FileSystem APIs to rename the file to the desired and delete the temp output folders created per date.

import org.apache.hadoop.conf.Configuration
import org.apache.spark.sql.SparkSession

import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
import org.apache.spark.SparkContext
import org.joda.time.format.DateTimeFormat


object MergeFiles {

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("Merging files day wise in a directory")
      .master("local[2]")
      .getOrCreate()

    val inputDir = "/Users/sujesh/test_data"
    val outputDir = "/Users/sujesh/output_data"

    val hadoopConf = spark.sparkContext.hadoopConfiguration
    val fs = FileSystem.get(hadoopConf)

    val filesPerDate = getFiles(inputDir, fs)

    filesPerDate
      .foreach { m =>
        spark
          .read
          .format("csv")
          .option("inferSchema", false)
          .option("header", false)
          .load(m._2:_*)
          .repartition(1)
          .write
          .format("csv")
          .save(s"$outputDir/${m._1}")

        val file = fs.globStatus(new Path(s"$outputDir/${m._1}/part*.csv"))(0).getPath.getName
        fs.rename(new Path(s"$outputDir/${m._1}/$file"), new Path(s"$outputDir/${m._1}.csv"))
        fs.delete(new Path(s"$outputDir/${m._1}"), true)
      }
  }

  /*
    Get the list of files group by date
    date is taken from file's modification timestamp
   */
  def getFiles(dir: String, fs: FileSystem) = {
    fs
      .globStatus(new Path(s"$dir/*.csv"))
      .map { f: FileStatus =>
        (DateTimeFormat.forPattern("yyyyMMdd").print(f.getModificationTime), f.getPath.toUri.getRawPath)
       }.groupBy(_._1)
       .map { case (k,v) => (k -> v.map(_._2).toSeq) }
  }
}

You may further optimise the code after tests and convert the file rename code to a util if it has to be re-used. Have put all options such as inferSchema or header to false. Use them as you need. This approach should work for other formats of files too.

Note: If you are repeatedly doing this process in the same directory, further tweaks would be required as the newly created files will have the latest timestamp. Hence if this is not run daily, you need to explicitly update the modification timestamp of the file too or ignore the files having a pattern of file name such as yyyyMMdd.csv