AWS S3数据湖跨账户使用

Question

we have the following scenario: AWS Account A (application) writes data from an application to an S3 bucket owned by account B (data lake). The analysts in account C (reporting) want to proccess the data and build reports and dashboards on top of it.

Account A can write data to the data lake with --acl bucket-owner-full-control to allow Account B the access. But Account C still cannot see and process the data.

One (in our eyes bad) solution is to copy the data to the same location (overwrite) as account B, effectively taking ownership for the data in the process and eliminating the issue. We don't want it, because ... ugly

We tried assuming roles in the different accounts, but it does not work for all our infrastructure. Eg S3 access via CLI or console is OK, but using it from EMR in account C does not. Also we have on-premise infrastructure (local taskrunners), where this mechanism is not an option.

Maintaining IAM roles for all accounts and users is too much effort. We aim for an automatic solution, not one that we have to take action every time a new user or account is added.

Do you have any suggestions?

Answer 1

You can do via the following documentation,

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-saml.html

Steps:

1. Create SAML provider
2. Create Role for the SAML provider, example below
3. Assign the users role based on saml conditions

Eg, You can create S3 Readers, S3 Writers and assign permissions based on that.

Example Assume Role with SAML:

{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {"Federated": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:saml-provider/ExampleOrgSSOProvider"},
        "Action": "sts:AssumeRoleWithSAML",
        "Condition": {"StringEquals": {
          "saml:edupersonorgdn": "ExampleOrg",
          "saml:aud": "https://signin.aws.amazon.com/saml"
        }}
      }]
}

Hope it helps.

Answer 2

One nice and clean way is to use a bucket policy granting read access to the external account (account C) by supplying the account ARN as the principal.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Grant read access to reporting account",
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:iam::insertReportingAccountIdHere:root"
         },
         "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl"
         ],
         "Resource": [
            "arn:aws:s3:::yourdatalakebucket",
            "arn:aws:s3:::yourdatalakebucket/*"
         ]
      }
   ]
}

This lets the reporting account manage the (ListBucket, gGtObject) permissions on the bucket for its own users, meaning you can now create an IAM policy on Account C with the permission to fetch data from the specified data lake bucket:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Allow reading files from the data lake",
         "Effect": "Allow",
         "Action": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl"
         ],
         "Resource": [
            "arn:aws:s3:::yourdatalakebucket",
            "arn:aws:s3:::yourdatalakebucket/*"
         ]
      }
   ]
}

This policy can then be attached to any Account C IAM role or user group you want. For example, you could attach it to your standard Developer or Analyst roles to give access to large groups of users, or you could attach it to a service role to give a particular service access to the bucket.

There is a guide on the Amazon S3 documentation site on how to do this.

Answer 3

In our case, we solved it using roles in the DataLake account (B), both for write (WriterRole) and read (ReaderRole) access. When writing to the DataLake from Account A, your writer assumes the "WriterRole" in Account B, that has the required permission. When reading from Account C, you assume the "ReaderRole". The issues with EMR reading, we solved with EMRFS using IAM roles for reading ( https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-emrfs-iam-roles.html )