Pyspark写入文件不存在

Question

I try to write in file in pyspark but I have an error, the file doesn't exist. I'm new in pyspark.

I have this code for write:

result.repartition(1).write.partitionBy('client', 'payload_type').json(OUTPUT_PATH, mode='append')

Is it possible to add a parameter to force to create folder/file if doesn't exist ?

Answer 1

I make a mistake, the error is not at the line of the post but this:

existing_data = spark.read.json(OUTPUT_PATH)

with:

OUTPUT_PATH = f"s3a://{BUCKET_DEST}/{SOURCE}/"

At the first execution, both folder doesn't exist, can I force create it if no exist on the read?