AWT IoT 规则到 S3

Question

I have created a rule to send the incoming IoT messages to a S3 bucket.

The problem is that any time IoT recieves a messages is sended and stored in a new file (with the same name) in S3.

I want this S3 file to keep all the data from before and not truncate each time a new message is stored.

How can I do that?

Answer 1

You can't do this with the S3 IoT Rule Action. You can get similar results using AWS Firehose, which will batch up several messages and write to one file. You will still end up with multiple files though.

Answer 2

You can do it using AWS IoT SQL variable expressions. For example use following as a key ${newuuid()} . This will create new s3 object for each message received.

See more about SQL Functions https://docs.aws.amazon.com/iot/latest/developerguide/iot-sql-functions.html

Answer 3

When you set up IoT S3 rule action , you need to specify a bucket and a key. The key is what we might think of as a "path and file name". As the docs say , can specify the key string by using a substitution template , which is just a fancy way of saying "build a path out of these pieces of information". When you are building your substitution template, you can reference fields inside the message as well as use use a bunch of other functions

Especially look at the functions topic , timestamp , as well as some of the string manipulator functions.

Let's say your topic names are something like things/thing-id-xyz/location and you just want to store each incoming JSON message in a "folder" for the thing-id it came in from. You might specify a key like:

${topic(2)}/${timestamp()).json

it would evaluate to something like:

thing-id-xyz/1481825251155.json

where the timestamp part is the time the message came in. That will be different for each message, and then the messages would not overwrite each other.

You can also specify parts of the message itself. Let's imagine our incoming messages look something like this:

{
  "time": "2022-01-13T10:04:03Z",
  "latitude": 40.803274,
  "longitude": -74.237926,
  "note": "Great view!"
}

Let's say you want to use the nice ISO date value you have in your data instead of the timestamp of the file. You could reference the time field no problem, like:

${topic(2)//${time}.json

Now the file would be written as the key:

thing-id-xyz/2022-01-13T10:04:03Z.json

You should be able to find some combination of values that works for your needs, and that most importantly, is UNIQUE for each message so they don't overwrite each other in S3.