使用Spark覆盖S3文件

Question

I have a use-case where after performing a join between two datasets, I need to write each row to a separate file (updating existing file) on S3. Does Spark support this?

If not, can we use S3 client explicitly to write each entry to a new file in S3? Are there any side-effects that I should be aware of?

Answer 1

it's not about Spark.. S3 doesn't support update, you should store the whole block at once.

in theory you could use multipart upload (MPU) to join multiple s3 object parts, however MPU is intended to support upload bigger than 5GB and minimal part size is 5MB

Each job can create new S3 object ( example )

Answer 2

As I said in my comment, tons of small files in S3 are usually a bad practice. That said, if you only have a limited amount of records to write, there are different options.

Here are some examples:

• use the DataFrameWriter, overwrite mode and partitionBy by a unique column
• use df.rdd.mapPartitions and write each records to S3 manually using hadoop S3FileSystem.

Good luck.