简体   繁体   English

Kinesis firehose 写入 S3 但访问被拒绝

[英]Kinesis firehose writes to S3 but access denied

  • I set up a Kinesis Firehose.我设置了一个 Kinesis Firehose。
  • Using a PUT data stream.使用 PUT 数据 stream。
  • It writes CSV data to S3.它将 CSV 数据写入 S3。
  • This is writing cross-account, from one AWS account to another one.这是写跨账户,从一个 AWS 账户到另一个账户。

This all works fine.这一切都很好。 I can download the data it writes to S3, I can query it.我可以下载它写入 S3 的数据,我可以查询它。

  • Another team created a different Firehose另一个团队创建了不同的 Firehose
  • This was in their AWS account, to write data to my S3 bucket.这是在他们的 AWS 账户中,用于将数据写入我的 S3 存储桶。
  • The data arrives, I can see it in the console.数据到达,我可以在控制台中看到它。
  • They are writing in Parquet format.他们以 Parquet 格式编写。

For some reason, I get "Access denied" if I try to download or query anything they have written to S3.出于某种原因,如果我尝试下载或查询他们写入 S3 的任何内容,我会收到“访问被拒绝”。

I sent data to the S3 bucket from my own firehose (in a third, different, account), and I can see that fine.我从我自己的 firehose(在第三个不同的帐户中)将数据发送到 S3 存储桶,我可以看到很好。 It's only the Parquet-format data from their Firehose that gets access denied .只有来自他们的 Firehose 的 Parquet 格式数据才会被拒绝访问

Troubleshooting故障排除

I compared the object-level permissions between the object from my firehose versus the object from their firehose: (Each of these come from different accounts into my account where the S3 bucket is).我比较了我的 firehose 的 object 和他们的 firehose 的 object 之间的对象级权限:(每个都来自不同的账户,进入我的 S3 存储桶所在的账户)。

  1. Here are the object permissions on the (CSV, text) object I firehosed into the S3 bucket:以下是我注入 S3 存储桶的(CSV,文本)object 的 object 权限:

These are the S3 objects I can read fine.这些是我可以正常阅读的 S3 对象。

Grantee : Object owner (external account) Canonical ID: 4aXXXXXXedc8fd受赠人:Object 所有者(外部帐户)规范 ID:4aXXXXXXedc8fd
Object : Read Object :读
Object ACL : Read, Write Object ACL :读、写

Grantee : Your AWS account Canonical ID: c43XXXXXXXX97958受赠人:您的 AWS 账户规范 ID:c43XXXXXXXX97958
Object : Read Object :读
Object ACL : Read, Write Object ACL :读、写

  1. Here are the Object permissions for the (Parquet JSON) object the other team firehosed into the bucket:以下是 (Parquet JSON) object 的 Object 权限,其他团队将其注入桶中:

These are the objects I get access denied for.这些是我被拒绝访问的对象。

These have exactly the same permissions, obviously the object owner account id is different because it is written from a firehose in a different account.它们具有完全相同的权限,显然 object 所有者帐户 ID 是不同的,因为它是从不同帐户的 firehose 中写入的。

Grantee : Object owner (external account) Canonical ID: 2efXXXXXXd5e2d受赠人:Object 所有者(外部帐户)规范 ID:2efXXXXXXd5e2d
Object : Read Object :读
Object ACL : Read, Write Object ACL :读、写

Grantee : Your AWS account Canonical ID: c43XXXXXXXX97958受赠人:您的 AWS 账户规范 ID:c43XXXXXXXX97958
Object : Read Object :读
Object ACL : Read, Write Object ACL :读、写

The problem:问题:

Why am I getting "Access denied" on the S3 objects the other team's Firehose created?为什么我在其他团队的 Firehose 创建的 S3 对象上收到“访问被拒绝”?

The object-level permissions look the same.对象级权限看起来是一样的。

We applied two fixes, and it worked我们应用了两个修复程序,并且有效

  1. The role in the target account that the Firehose uses, needs to have S3:PutObjectAcl granted. Firehose 使用的目标帐户中的角色需要授予 S3:PutObjectAcl。 This was checked early on but after re-checking it had gone, so this was added back.这是早期检查过的,但在重新检查后它已经消失了,所以它被添加回来了。

  2. We turned off KMS encryption in the sending account.我们在发送帐户中关闭了 KMS 加密。 This was using a key which the target account could not access.这是使用目标帐户无法访问的密钥。 This could be fixed by adding permissions to the role that writes the data, but for debugging we stripped everything back to get it to work as a baseline.这可以通过向写入数据的角色添加权限来解决,但为了调试,我们剥离了所有内容以使其作为基线工作。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 Kinesis Firehose 写入 S3 云监视订阅过滤器,但文件不可读 - Kinesis Firehose writes to S3 cloud watch subscription filter but the files are not readable Kinesis Firehose 将 JSON 个对象放入 S3 中,没有分隔符逗号 - Kinesis Firehose putting JSON objects in S3 without seperator comma 以 Kinesis Firehose output 格式将 DynamoDB 数据传输到 S3 - DynamoDB data to S3 in Kinesis Firehose output format 按事件时间对 Kinesis firehose S3 记录进行分区 - Partition Kinesis firehose S3 records by event time 读取 Amazon Kinesis Firehose 写入 s3 的数据 stream - Reading the data written to s3 by Amazon Kinesis Firehose stream 使用来自 Kinesis Data Stream 源的 Kinesis Firehose Delivery Stream 将数据写入 S3 时出现问题 - Problem writing data to S3 with Kinesis Firehose Delivery Stream from Kinesis Data Stream source Kinesis Firehose 将数据从 DynamoDB Steam 传输到 S3:为什么文件中的 JSON 个对象数量不同? - Kinesis Firehose delivers data from DynamoDB Steam to S3: Why the numbers of JSON objects in files is different? 无法触发由 Kinesis Firehose 传输流创建的 S3 对象上的事件 - Unable to trigger event on S3 object created by Kinesis Firehose delivery stream AWS S3 Java SDK - 拒绝访问 - AWS S3 Java SDK - Access Denied 关于复制 S3“CopyObject 操作:访问被拒绝” - On Copying S3 "CopyObject operation: Access Denied"
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM