从 S3 存储桶读取的 AWS Glue 限制数据

Question

I have a large bucket that contains more than 6M files. I've run into this error Failed to sanitize XML document destined for handler class and i think this is the problem: https://github.com/lbroudoux/es-amazon-s3-river/issues/16

Is there a way I can limit how many files are read in the first runs?

This is what I have DataSource0 = glueContext.create_dynamic_frame.from_catalog(database = "s3-sat-dth-prd", table_name = "datahub_meraki_user_data", transformation_ctx = "DataSource0") , can I tell it to read only a folder in my bucket? Every folder within is called like this: partition=13/ , partition=14/ , partition=n/ and so on.

How can I work around this?

Thanks in advance.

Answer 1

There are three main ways (as I know) to avoid this situation.

1. Load from a prefix

In order to load files from a specific path in AWS Glue, you can use the below syntax.

from awsglue.dynamicframe import DynamicFrame

dynamic_frame = context.create_dynamic_frame_from_options(
        "s3",
        {
            'paths': ['s3://my_bucket_1/my_prefix_1'],
            'recurse': True,
            'groupFiles': 'inPartition',
            'groupSize': '1073741824'
        },
        format='json',
        transformation_ctx='DataSource0'
    )

You can put multiple paths for paths and Glue will load from all of them.

2. Use Glue Bookmarks.

When you have millions of files in a bucket and you want to load only the new files (between the runs of your Glue job), you can enable Glue Bookmarks. It will keep track of the files it read in an internal index (which we don't have access to). You can pass this as a parameter when you define the job.

  MyJob:
    Type: AWS::Glue::Job
    Properties:
      ...
      GlueVersion: 2.0
      Command:
        Name: glueetl
        PythonVersion: 3
        ...
      DefaultArguments: {
        "--job-bookmark-option": job-bookmark-enable,
        ...

This will enable bookmarks defined with the name used for transformation_ctx when you load data. Yes, it's confusing that AWS uses the same parameter for multiple purposes!

It's also important that you must not forget to add job.commit() at the end of your Glue script, where job is your from awsglue.job import Job instance.

Then, when you use the same context.create_dynamic_frame_from_options() function with your root prefix and the same transformation_ctx , it will only load the new files in the prefix in the hierarchy. It saves a lot of hassle for us in looking for new files. Read the docs for more information on bookmarks.

3. Avoid smaller file sizes.

AWS Glue will take ages to load files if you have quite smaller files. So, if you can control the file size, then make the files at least 100MB in size. For instance, we were writing to S3 from a Firehose stream and we could adjust the buffer size to avoid smaller file sizes. This drastically increased the loading times for our Glue job.

I hope these tips will help you. And feel free to ask any questions if you need further clarification.

Answer 2

There is a way to control the # of files called a BoundedExecution. It's documented here: https://docs.aws.amazon.com/glue/latest/dg/bounded-execution.html

In the following examples you would be loading in 200 files at a time. Note you must enable Glue bookmarks for this to work correctly.

If you are using from_options it looks like this:

    DataSource0 = glueContext.create_dynamic_frame.from_options(
        format_options={"withHeader": True, "separator": separator, "quoteChar": quoteChar},
        connection_type="s3",
        format="csv",
        connection_options={"paths": inputFilePath,
                            "boundedFiles": "200", "recurse": True},
        transformation_ctx="DataSource0"
    )

If you are using from_catalog it looks like this:

    DataSource0 = glueContext.create_dynamic_frame.from_catalog(
        database = "database-name",
        table_name= "table-name",
        additional_options={"boundedFiles": "200"},
        transformation_ctx="DataSource0"
    )