AWS Glue 作业以 Parquet 格式错误写入 s3，但未找到

Question

I've been creting pyspark jobs and I keep getting one similar and intermittently error (is more like random):

An error occurred while calling o129.parquet. Not Found 
(Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; 
Request ID: D2FA355F92AF8F05; S3 Extended Request ID: 1/fWdf1DurwPDP40HDGARlMRO/7lKzFDJ4g7DbUnM04wUvG89CG9w5T+u4UxapkWp20MfQfdjsE=)

I'm not even reading from s3, what I'm actually doing is: df.coalesce(100).write.partitionBy("mth").mode("overwrite").parquet("s3://"+bucket+"/"+path+"/out")

So I change the coalesce partition, but I'm not wure what else should I do to mitigate this error and make my jobs more stable.

Answer 1

for reading the file from s3 using glue

datasource0 = glueContext.create_dynamic_frame.from_options( connection_type = "s3", connection_options = {"paths": "s3/path"}, format = "json", transformation_ctx = "datasource0")

for writing file to s3 using glue

output = glueContext.write_dynamic_frame.from_options(frame = df, connection_type = "s3", connection_options = {"path": "s3/path"}, format = "parquet", transformation_ctx = "output")