Sending Data From AWS IoT To Databases

Question

I have a question for you. I am new in AWS IoT, AWS ES and MQTT. I followed this nice tutorial and finished it. I sent a JSON data to AWS IoT by using AWS Lambda function. Then, in AWS IoT I created a rule to send this data to AWS Elasticsearch. In the end, I visualized the data by using Kibana.

My next task is storing this data in 3 different types of databases: Relational DB (Amazon Aurora), Key Value DB (Amazon Dynamo DB) and Document DB (Amazon Document DB). But as you know, AWS IoT doesn't provide these options as a rule. It only provides "Insert A Message Into DynamoDB Table" option. In that case, how can I create other databases and send data from AWS IoT? Is there any source or tutorial that you can suggest for me about that?

I would be very happy if experienced people can guide me Thanks a lot!

(FYI: My elasticsearch doesn't have a VPC. I am using public access. So far, I didn't use any EC2.)

Answer 1

The most common method of inserting AWS IoT data into a database is to configure IoT to send the messages to an AWS Lambda function. Inside the Lambda function you would connect to your database and insert the data just like any other database interaction.

Answer 2

You can use AWS IoT Analytics to store messages in an S3 bucket. A simple pipeline (with no transformations) will store incoming messages in a compressed format, and the output in json.gz files named by data/timestamp. If you configure you IoT Analytics with custom S3 Buckets, you can set the retention policy of the buckets to expire old data.

The you can write some code to read the Analytics Pipeline output files and put them into whatever data store you want.

Or you can just send the messages to a Lambda function and have them connect and store messages in a database. The advantage of using an Analytics Pipeline is you can continue to recieve/store MQTT messages if you've got database downtime, or you deploy database ingestion code with bugs.