无法使用 spark 在 S3 上创建分区

Question

I would like to use this new functionality: overwrite specific partition without delete all data in s3

I used the new flag ( spark.sql.sources.partitionOverwriteMode="dynamic" ) and test it locally from my IDE and it worked (I was able to overwrite specific partition in s3) but when I deployed it to hdp 2.6.5 with spark 2.3.0 same code didn't create the s3 folders as expected , folder didn't create at all , only temp folder has been created

My code :

df.write
.mode(SaveMode.Overwtite)
.partitionBy("day","hour")
.option("compression", "gzip")
.parquet(s3Path)

Answer 1

Have you tried spark version 2.4 ? I have worked with this version and both EMR and Glue it has worked well, to use the "dynamic" in version 2.4 just use the code:

dataset.write.mode("overwrite")
.option("partitionOverwriteMode", "dynamic")
.partitionBy("dt")
.parquet("s3://bucket/output")

AWS documentation specifies Spark version 2.3.2 to use spark.sql.sources.partitionOverwriteMode="dynamic" .

Reference click here .