AWS Firehose跨区域/帐户策略

Question

I am trying to create Firehose streams that can receive data from different regions in Account A, through AWS Lambda, and output into a redshift table in Account B. To do this I created an IAM role on Account A:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "firehose.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

I gave it the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::b-bucket/*",
                "arn:aws:s3:::b-bucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "firehose:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "redshift:*"
            ],
            "Resource": "*"
        }
    ]
}

On Account BI created a role with this trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "firehose.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "11111111111"
                }
            }
        }
    ]
}

I gave that role the following access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::b-bucket",
                "arn:aws:s3:::b-bucket/*",
                "arn:aws:s3:::b-account-logs",
                "arn:aws:s3:::b-account-logs/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "firehose:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "redshift:*",
            "Resource": "arn:aws:redshift:us-east-1:cluster:account-b-cluster*"
        }
    ]
}

I also edited the access policy on the S3 buckets to give access to my Account A role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::11111111111:role/AccountAXAccountBPolicy"
            },
            "Action": "s3:*",
            "Resource": ["arn:aws:s3:::b-bucket","arn:aws:s3:::b-bucket/*"]
        }
    ]
}

However, none of this works. When I try to create the the stream in Account A it does not list the buckets in Account B nor the redshift cluster. Is there any way to make this work?

Answer 1

John's answer is semi correct. I would recommend that the account owner of the Redshift Cluster creates the FireHose Stream. Creating through CLI requires you to supply the user name and password. Having the cluster owner create the stream and sharing IAM Role permissions on the stream is safer for security and in case of credential change. Additionally, you cannot create a stream that accesses a database outside of the region, so have the delivery application access the correct stream and region.

Read on to below to see how to create the cross account stream.

In my case both accounts are accessible to me and to lower the amount of changes and ease of monitoring I created the stream on Account A side.

The above permissions are right however, you cannot create a Firehose Stream from Account A to Account B through AWS Console. You need to do it through AWS Cli:

 aws firehose create-delivery-stream --delivery-stream-name testFirehoseStreamToRedshift 
 --redshift-destination-configuration 'RoleARN="arn:aws:iam::11111111111:role/AccountAXAccountBRole", ClusterJDBCURL="jdbc:redshift://<cluster-url>:<cluster-port>/<>",
 CopyCommand={DataTableName="<schema_name>.x_test",DataTableColumns="ID1,STRING_DATA1",CopyOptions="csv"},Username="<Cluster_User_name>",Password="<Cluster_Password>",S3Configuration={RoleARN="arn:aws:iam::11111111111:role/AccountAXAccountBRole",
 BucketARN="arn:aws:s3:::b-bucket",Prefix="test/",CompressionFormat="UNCOMPRESSED"}'

You can test this by creating a test table on the other AWS Account:

create table test_schema.x_test
(
    ID1 INT8 NOT NULL,
    STRING_DATA1 VARCHAR(10) NOT NULL
)
distkey(ID1)
sortkey(ID1,STRING_DATA1);

You can send test data like this:

aws firehose put-record  --delivery-stream-name testFirehoseStreamToRedshift --record '{"DATA":"1,\"ABCDEFGHIJ\""}'

This with the permissions configuration above should create the cross account access for you.

Documentation:
Create Stream - http://docs.aws.amazon.com/cli/latest/reference/firehose/create-delivery-stream.html

Put Record - http://docs.aws.amazon.com/cli/latest/reference/firehose/put-record.html

Answer 2

No.

Amazon Kinesis Firehose will only output to Amazon S3 buckets and Amazon Redshift clusters in the same region.

However, anything can send information to Kinesis Firehose by simply calling the appropriate endpoint. So, you could have applications in any AWS Account and in any Region (or anywhere on the Internet) send data to the Firehose and then have it stored in a bucket or cluster in the same region as the Firehose.