由于 IAM 权限，无法运行 AWS Glue Crawler

Question

I am unable to run newly created AWS Glue Crawler. I followed IAM Role guide at https://docs.aws.amazon.com/glue/latest/dg/create-an-iam-role.html?icmpid=docs_glue_console

1. Created new Crawler Role AWSGlueServiceRoleDefault with AWSGlueServiceRole and AmazonS3FullAccess managed policies
2. Trust Relationship contains:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "glue.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

3. User executing crawler signs via SSO and inheriths arn:aws:iam::aws:policy/AdministratorAccess
4. I even tried to create new AWS user with all permissions

After executing Crawler it fails within 8 seconds with following error:

Crawler cannot be started. Verify the permissions in the policies attached to the IAM role defined in the crawler

What other IAM permissions are needed?

Answer 1

If you're crawling tables and schemas via a JDBC connection to an external data store, make sure you have specified.network options to the Glue Connection. I got the exactly same error if the options is not specified. I think the error message is somewhat misleading here.

Here's what I have defined to my crawlers:

1. A role, eg AWSGlueServiceRoleDefault with AWSGlueServiceRole managed policies attached.

2. Specify the.network options to your connections.

3. A NAT gateway is created and attached to the su.net you have defined in the step 2 so that there is a public IP available for your crawler to connect to the external data store.

If you're attempting to connecting RDS, since the crawler and the database are both in the AWS.network, a NAT is not needed. Just define the security group rules to allow the connections. Check the document here .

If S3 is the target data source, a VPC endpoint for S3 is recommended. Check the document here .