如果Spark不存在，Spark会创建一个s3文件夹路径吗？

Question

Let's say I have this line, I want to know if Spark automatically creates a folder path and writes to folder like it does in local.

Yes, s3 is not folder system rather a key val system.

val path="s3a://dev-us-east-1/"

val op = df_formatted.coalesce(1).write.mode("overwrite").format("csv").save(path + "report/output")

Will this be written to "s3a://dev-us-east-1/report/output"

Answer 1

如果您已正确设置了“安全性”，即您拥有具有写访问权的IAM用户的凭据，那么Spark将创建文件夹和文件。