Spark 中压缩镶木地板文件的扩展

Question

In my Spark job, I write a compressed parquet file like this:

df
  .repartition(numberOutputFiles)
  .write
  .option("compression","gzip")
  .mode(saveMode)
  .parquet(avroPath)

Then, my files has this extension: file_name .gz.parquet

How can I have ".parquet.gz" ?

Answer 1

I don't believe you can. File extension is hardcoded in ParquetWrite.scala as concatenation of codec + ".parquet":

  :
    override def getFileExtension(context: TaskAttemptContext): String = {
      CodecConfig.from(context).getCodec.getExtension + ".parquet"
    }
  :