How do I limit access to S3 Bucket for particular IAM Role?

Question

We want to store some data on S3 and only allow EC2 instances or a particular user with a particular IAM role to access them. Unfortunately we're having some trouble doing this.

We set a policy on the bucket like this

{
"Version": "2012-10-17",
"Id": "SamplePolicy",
"Statement": [

    {
        "Sid": "Stmt1331136294179",
        "Effect": "Deny",
        "NotPrincipal": {
            "AWS": [
                "arn:aws:iam::our-account-number:user/the-user",

                "arn:aws:iam::our-account-number:role/the-role"
            ]
        },
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::the-bucket/*"
    },
        {
        "Sid": "Stmt1331136364169",
        "Effect": "Allow",
        "Principal": {
            "AWS": [
                "arn:aws:iam::our-account-number:user/the-user",
                "arn:aws:iam::our-account-number:role/the-role"
            ]
        },
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::the-bucket/*"
    }

]}

When we access the Bucket (using boto) with users key it works fine, from a local machine or any EC2 instance.

But, when we access the bucket from Boto we get

ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

I've confirmed that the instance has the correct IAM role

   curl http://169.254.169.254/latest/meta-data/iam/info/
{
  "Code" : "Success",
  "LastUpdated" : "2015-10-22T09:09:31Z",
  "InstanceProfileArn" : "our-account-number:instance-profile/the-role",
  "InstanceProfileId" : "instance-rpofile-id"
}

I've also tried to remove the policy from the bucket, which indeed makes it accessible again. Any ideas how to handle this?

The sample I shared here is a simplified version I've been doing for debugging. In production, we want are forcing the object to be encrypted with KMS and have an access policy on the key as well. We like that solution alot, and prefer to keep it if we can.

Thanks

Answer 1

One mistake with this that I've made many times involves your ARN

For some permissions you need it on the bucket itself (no /*)... and some you need on it's contents.

I'd attempt to use what you currently have, only include both, so something like...

"Resource": ["arn:aws:s3:::the-bucket/*", "arn:aws:s3:::the-bucket"]

Answer 2

The issue here is that for NotPrincipal you have to provide the specific session role. Unfortunately, when using InstanceProfiles (or Lambda), this session role is dynamic. AWS does not support wildcards in the principal field so therefore it is basically impossible to use NotPrincipal with an InstanceProfile.

See AWS support response here that acknowledges it as a known limitation: https://forums.aws.amazon.com/message.jspa?messageID=740656#740656