AWS Glue 作业在写入 S3 时被拒绝访问

Question

I have a Glue ETL job, created by CloudFormation. This job extracts data from RDS Aurora and write to S3.

When I run this job, I get the error below.

The job has an IAM service role.

This service role allows

1. Glue and RDS service,
2. assume arn:aws:iam::aws:policy/AmazonS3FullAccess and arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole, and
3. has full range of rds:*, kms:*, and s3:* actions allow to the corresponding RDS, KMS, and S3 resources.

I have the same error whether the S3 bucket is encrypted with either AES256 or aws:kms.

I get the same error whether the job has a Security Configuration or not.

I have a job doing the exactly same thing that I created manually and can run successfully without a Security Configuration.

What am I missing? Here's the full error log

"/mnt/yarn/usercache/root/appcache/application_1...5_0002/container_15...45_0002_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o145.pyWriteDynamicFrame. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 2.0 failed 4 times, most recent failure: Lost task 3.3 in stage 2.0 (TID 30, ip-10-....us-west-2.compute.internal, executor 1): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: F...49), S3 Extended Request ID: eo...wXZw= at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588

Answer 1

Unfortunately the error doesn't tell us much except that it's failing during the write of your DynamicFrame.

There is only a handful of possible reasons for the 403, you can check if you have met them all:

1. Bucket Policy rules on the destination bucket.
2. The IAM Role needs permissions (although you mention having S3*)
3. If this is cross-account, then there is more to check with regards things like to allow-policies on the bucket and user. (In general a Trust for the Canonical Account ID is simplest)
4. I don't know how complicated your policy documents might be for the Role and Bucket, but remember that an explicit Deny statement takes precedence over an allow.
5. If the issue is KMS related, I would check to ensure your Subnet you select for the Glue Connection has a route to reach the KMS endpoints (You can add an Endpoint for KMS in VPC)
6. Make sure issue is not with the Temporary Directory that is also configured for your job or perhaps write-operations that are not your final.
7. Check that your account is the "object owner" of the location you are writing to (normally an issue when read/writing data between accounts)

If none of the above works, you can shed some more light with regards to your setup. Perhaps the code for write-operation.

Answer 2

In addition to Lydon's answer, error 403 is also received if your Data Source location is the same as the Data Target; defined when creating a Job in Glue. Change either of these if they are identical and the issue will be resolved.

Answer 3

How are you providing permission for PassRole to glue role?

{
        "Sid": "AllowAccessToRoleOnly",
        "Effect": "Allow",
        "Action": [
          "iam:PassRole",
          "iam:GetRole",
          "iam:GetRolePolicy",
          "iam:ListRolePolicies",
          "iam:ListAttachedRolePolicies"
        ],
        "Resource": "arn:aws:iam::*:role/<role>"
      }

Usually we create roles using <project>-<role>-<env> eg xyz-glue-dev where project name is xyz and env is dev. In that case we use "Resource": "arn:aws:iam::*:role/xyz-*-dev"

Answer 4

For me it was two things.

1. Access policy for a bucket should be given correctly - bucket/*, here I was missing the * part
2. Endpoint in VPC must be created for glue to access S3 https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoints-s3.html

After these two settings, my glue job ran successfully. Hope this helps.

Answer 5

Make sure you have given the right policies. I was facing the same issue, thought I had the role configured well. But after I erased the role and followed this step, it worked ;]

Answer 6

You should add a Security configurations(mentioned under Secuity tab on Glue Console). providing S3 Encryption mode either SSE-KMS or SSE-S3.

Security Configuration

Now select the above security configuration while creating your job under Advance Properties.

Duly verify you IAM role & S3 bucket policy. It will work

Answer 7

Have you done the required RDS Parameter group configurations, I dont see any reference to this in your question? I am assumiing you have missed certain configurations in the RDS Parameter groups to read/write from S3 & RDS. If this is not done, please refer to this link and do required configurations. This should work.