[英]How to read data from Kinesis stream using AWS CLI?
I have a Kinesis stream in AWS and can send data to it (JSON) using kinesis command and can get it back from a stream with:我在 AWS 中有一个 Kinesis 流,可以使用 kinesis 命令向它发送数据(JSON),并且可以通过以下方式从流中取回:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name mystream --query 'ShardIterator' --profile myprofile)
aws kinesis get-records --shard-iterator $SHARD_ITERATOR --profile myprofile
Output of this looks like something like:这个输出看起来像这样:
HsKCQkidmlkZW9Tb3VyY2UiOiBbCgkJCXsKCQkJCSJicmFuZGluZyI6IHt9LAoJCQkJInByb21vUG9vbCI6IFtdLAoJCQkJImlkIjogbnVsbAoJCQl9CgkJXSwKCQkiaW1hZ2VTb3VyY2UiOiB7fSwKCQkibWV0YWRhdGFBcHByb3ZlZCI6IHRydWUsCgkJImR1ZURhdGUiOiAxNTgzMzEyNTA0ODAzLAoJCSJwcm9maWxlIjogewoJCQkiY29tcG9uZW50Q291bnQiOiAwLAoJCQkibmFtZSI6ICJTUUVfQVRfUFJPRklMRSIsCgkJCSJpZCI6ICJTUUVfQVRfUFJPRklMRV9JRCIsCgkJCSJwYWNrYWdlQ291bnQiOiAwLAoJCQkicGFja2FnZXMiOiBbCgkJCQl7CgkJCQkJIm5hbWUiOiAiUEVBQ09DSy1MVEEiLAoJCQkJCSJpZCI6ICJmZDk5NTRmZC03NDYwLTRjZjItOTU5Ni05YzBhMjcxNTViODgiCgkJCQl9CgkJCV0KCQl9LAoJCSJ3b3JrT3JkZXJJZCI6ICJTUUVfQVRfSk9CX1NVQk1JU1
How do I get actual JSON message in raw format (to look as JSON) - same way as it was in original when I sent it?我如何以原始格式获取实际的 JSON 消息(看起来像 JSON) - 与我发送时的原始格式相同?
Thanks谢谢
As per the docs , you need to use a Base64 decoding tool or use KCL library to get the data in the format it was sent:根据文档,您需要使用 Base64 解码工具或使用 KCL 库以发送的格式获取数据:
The first thing you'll likely notice about your record in this part of the tutorial is that the data appears to be garbage –;
在本教程的这一部分中,您可能会注意到有关您的记录的第一件事是数据似乎是垃圾——; it's not the clear text testdata we sent.
这不是我们发送的明文测试数据。 This is due to the way put-record uses Base64 encoding to allow you to send binary data.
这是由于 put-record 使用 Base64 编码的方式来允许您发送二进制数据。 However, the Kinesis Data Streams support in the AWS CLI does not provide Base64 decoding because Base64 decoding to raw binary content printed to stdout can lead to undesired behavior and potential security issues on certain platforms and terminals.
但是,AWS CLI 中的 Kinesis Data Streams 支持不提供 Base64 解码,因为对打印到 stdout 的原始二进制内容进行 Base64 解码可能会导致某些平台和终端上出现意外行为和潜在安全问题。 If you use a Base64 decoder (for example, https://www.base64decode.org/ ) to manually decode dGVzdGRhdGE= you will see that it is, in fact, testdata.
如果您使用 Base64 解码器(例如https://www.base64decode.org/ )手动解码 dGVzdGRhdGE=,您将看到它实际上是测试数据。 This is sufficient for the sake of this tutorial because, in practice, the AWS CLI is rarely used to consume data, but more often to monitor the state of the stream and obtain information, as shown previously (describe-stream and list-streams).
对于本教程而言,这已经足够了,因为在实践中,AWS CLI 很少用于使用数据,但更常用于监控流的状态和获取信息,如前文所示(describe-stream 和 list-streams) . Future tutorials will show you how to build production-quality consumer applications using the Kinesis Client Library (KCL), where Base64 is taken care of for you.
未来的教程将向您展示如何使用 Kinesis 客户端库 (KCL) 构建生产质量的消费者应用程序,其中为您处理 Base64。 For more information about the KCL, see Developing KCL 1.x Consumers.
有关 KCL 的更多信息,请参阅开发 KCL 1.x 使用者。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.