将 AWS IoT 规则连接到单独 AWS 账户中的 MSK Kafka 实例时出现问题

Question

Thank you in advance for any help!

My company uses multiple AWS accounts to separate region and "global" functionality/storage. We've recently switched to using MQTT IoT Rules and have run tests with regional Kafka vs a global Kafka instance for how we process data, deciding that a global Kafka makes the most sense.

I've been able to properly setup a regional Kafka with the regional IoT pushing data to it (in the same account), but when I tried setting up a global Kafka with the regional IoT pushing data it failed (separate accounts).

The IoT Rule also does not post any errors to Cloudwatch Logs, despite setting up a log group as an error action.

We've set everything up to use peer connections and allowing CIDR access to security groups and have many other services running from region to global just fine, but this one is failing us.

I've even been able to create an EC2 instance in the region under the same VPC, Su.net, and Security group as the ones I've assigned to my IoT Rule and it's working just fine to produce and consume from Kafka.

The only place I can imagine the issue being is with the bootstrap URLs not being accessible inside the region or the secret username/password not having proper access. The secret has been set in the global account. The role that I'm using has been given permission to read that secret in the separate account.

arn:aws:iam::12345:role/TestRole:

{
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:region:9876:secret:AmazonMSK_TestSecret"
        }

${get_secret('arn:aws:secretsmanager:region:9876:secret:AmazonMSK_TestSecret', 'SecretString', 'username', 'arn:aws:iam::12345:role/TestRole')}

If anyone is familiar with reading secrets across accounts or using MSK Bootstrap URLs across accounts I'd appreciate some insight.

Answer 1

AWS IoT rules supports cross account but only for a limited list of services: Amazon SQS, Amazon SNS, Amazon S3, and AWS Lambda.

You can see this information here .

So in your case, you would have to push data into a SQS queue in the other account and then pull data using Lambda in order to ingest it to MSK.

Answer 2

Like mentioned before it's not supported by AWS..

We solved it by using MirrorMaker2 by strimzi to replicate events between clusters across account (requires a VPC peering or any kind of.network connectivity solution)