将 Spark DF 保存到 s3 路径是否首先将数据写入 EBS 卷？

Question

I am curious to know what happens behind the scenes when writing Spark DF as a Parquet file on S3 location. Does it first stores it locally on the local file system(EBS in our case) and then pushes those files to S3 finally?. Once the local files are successfully pushed, Are the local files on the EBS volume deleted?.

If it stores locally on EBS volume, what is the path it chooses to write the local files?. What is the spark configuration property that sets this path?.

Sample code that saves DF to S3 location:

df.repartition(prtn_col[0]).write.format(self.config_dict['output_format']).mode('overwrite').partitionBy(
                *prtn_col).save(path)

Software Versions used

1. Python version: 3.7.0
2. PySpark version: 2.4.7
3. Emr: 5.32.0

Please let me know if you like to me to share any other info that will help answering this question.

Answer 1

If you are using S3 URIs that look like s3://bucket/path on EMR (as opposed to s3a://bucket/path), you are using a component called EMRFS, EMR's closed source implementation of the S3 filesystem connector. When writing files to S3 (including via Spark DataFrame write operations), EMRFS does write the file first to temporary files on the local filesystem. And yes, the temporary files are deleted upon completing the upload (or at some point in the future, such as when the YARN container running the Spark executor is terminated, if the S3 upload does not complete normally).

The temporary files might be written to EBS, but it depends upon what types of volumes are attached to the cluster nodes that are performing the writes. Many instance types are "EBS-only", meaning that they don't have any of their own storage (such as SSDs) and require you to attach EBS volumes. (If left unspecified, EMR will also automatically add a certain number of EBS volumes by default to EBS-only instance types.)