AccessDenied error from CreateBucket Permissions for pandas to_csv to S3

Question

I have a script running on an EC2 box that finishes by running pd.to_csv('s3://<my_bucket_name>/<file_path> .

Run locally with my AWS admin credentials, this script runs fine and deposits the csv into the right bucket.

My S3 permissions for the EC2 instance are copied and pasted out of AWS' documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::<my_bucket_name>"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object*",
            "Resource": ["arn:aws:s3:::<my_bucket_name>/*"]
        }
    ]
}

When run on the EC2 instance, my error is botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied .

I don't understand why pandas/s3fs is trying to create a bucket when mine already does exist. Suggestions elsewhere was to just provide s3:* access to ec2, but I'd prefer to be a little more restrictive than no restrictions.

Any thoughts on how to resolve this?

Answer 1

Turns out this was more of an issue with The aws batch role that was running the ec2 instance. The write permissions are good enough to write to S3 without bucket listing privileges. The AccessDenied error was a red herring at the more general error that no privileges were being passed to the instance.

Answer 2

A quick look at the Pandas codebase didn't show me anything concrete, but my guess would be that it's checking to see if the bucket exists before listing/updating the objects and failing because it doesn't have the s3:ListAllMyBuckets permission .

You could confirm or deny this theory by giving your role that action (in its own statement), which would hopefully avoid having to give s3:* to it.