如何使用 Spark/Scala 在 HDFS 上编写/创建 zip 文件？

Question

I need to write a Spark/Scala function in Apache Zeppelin that simply puts some files that are already present in an HDFS folder into a zip or gzip archive (or some common archive format that is easy to extract in Windows) in the same folder. How would I do this please? Would it be a Java call? I see there's something called ZipOutputStream, is that the right approach? Any tips appreciated.

Thanks

Answer 1

Spark does not support reading/writing from zip directly, so using the ZipOutputStream is basically the only approach.

Here's the code I used to compress my existing data via spark. It recursively lists directory for files and then proceeds to compress them. This code does not preserve directory structure, but keeps file names.

Input directory:

unzipped/
├── part-00001
├── part-00002
└── part-00003

0 directories, 3 files

Output directory:

zipped/
├── part-00001.zip
├── part-00002.zip
└── part-00003.zip

0 directories, 3 files

ZipPacker.scala:

package com.haodemon.spark.compression

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.io.IOUtils
import org.apache.spark.sql.SparkSession
import org.apache.spark.{SparkConf, SparkContext}

import java.io.FileOutputStream
import java.util.zip.{ZipEntry, ZipOutputStream}


object ZipPacker extends Serializable {

  private def getSparkContext: SparkContext = {
    val conf: SparkConf = new SparkConf()
      .setAppName("local")
      .setMaster("local[*]")
    SparkSession.builder().config(conf).getOrCreate().sparkContext
  }

  // recursively list files in a filesystem
  private def listFiles(fs: FileSystem, path: Path): List[Path] = {
    fs.listStatus(path).flatMap(p =>
        if (p.isDirectory) listFiles(fs, p.getPath)
        else List(p.getPath)
      ).toList
  }

  // zip compress file one by one in parallel
  private def zip(inputPath: Path, outputDirectory: Path): Unit = {
    val outputPath = {
      val name = inputPath.getName + ".zip"
      outputDirectory + "/" + name
    }
    println(s"Zipping to $outputPath")

    val zipStream = {
      val out = new FileOutputStream(outputPath)
      val zip = new ZipOutputStream(out)
      val entry = new ZipEntry(inputPath.getName)
      zip.putNextEntry(entry)
      // max compression
      zip.setLevel(9)
      zip
    }

    val conf = new Configuration
    val uncompressedStream = inputPath.getFileSystem(conf).open(inputPath)
    val close = true
    IOUtils.copyBytes(uncompressedStream, zipStream, conf, close)
  }

  def main(args: Array[String]): Unit = {
    val input = new Path(args(0))
    println(s"Using input path $input")

    val sc = getSparkContext
    val uncompressedFiles = {
      val conf = sc.hadoopConfiguration
      val fs = input.getFileSystem(conf)
      listFiles(fs, input)
    }
    val rdd = sc.parallelize(uncompressedFiles)

    val output = new Path(args(1))
    println(s"Using output path $output")

    rdd.foreach(unzipped => zip(unzipped, output))
  }
}