AWS Glue - 防止空导出到 S3

Question

How can I avoid that AWS Glue writes empty objects to S3?

I have a Glue Job that writes the resulting dynamic frame to S3:

dynamic_frame = # result of Glue job processing

glue_context.write_dynamic_frame.from_options(
    frame = dynamic_frame,
    connection_type = 's3',
    connection_options = {'path': 's3://some-bucket/some-path'},
    format = 'json')

However, when I check the bucket content in S3, I see not just the data but also many objects that has size 0 B. How can I prevent this?

I have tried using the DropNullFields class (see below), but that did not help.

dynamic_frame = # result of Glue job processing

non_null_fields = DropNullFields.apply(dynamic_frame)

glue_context.write_dynamic_frame.from_options(
    frame = non_null_fields,
    connection_type = 's3',
    connection_options = {'path': 's3://some-bucket/some-path'},
    format = 'json')

Answer 1

Aws Glue is a wrapper around Apache Spark. Normally Spark writes as many files as partitions. If it is writing empty files it means that you have empty partitions. The way to do it would be to repartition your dynamic_frame in smaller partitions. With Spark DataFrames you would use the "coalesce" function.

In Glue you could try to use the repartition function: https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-apis-glue-dynamicframe-class.html#glue-etl-scala-apis-glue-dynamicframe-class-defs-repartition