AWS：如何将流式数据保存到托管在 EC2 上的数据库（例如 MySQL/MongoDB）

Question

We can easily save data between different AWS Services for ex. Kinesis to DynamoDB; or AWS IoT to Redshift etc.

But what is best strategy to save streaming data to suppose MongoDB ( which does NOT have AWS PaaS; Atlas is there but it has no integrations with other AWS Services )

I can see some third party solutions are there; but what is best strategy to implement on AWS itself...Is execution of lambda function for each insert (batching) the only option?

Answer 1

I am assuming that you are using Kinesis Firehose. If that's the case, what you can do is:

• From Firehose write to S3 every 5 mins.

• Firehose will create a new file on S3 every 5 mins.

• Trigger a Lambda function to read the new file on S3.

• Write the data of the new file to MongoDB.

If you are using Kinesis (not firehose), you can simply write a Kinesis consumer which will read data from the Kinesis and write directly yo MongoDB.

FYI, There is DocumentDB with MongoDB like API, you can use that as AWS Hosted MongoDB

Answer 2

You can invoke lambda function on each FireHose invocation. And this lambda can insert into mongodb hosted on EC2. You can batch operations so as to reduce number of lambda invocations ( and in return reduce cost )

Answer 3

The solution depends mostly on your use case. How fast do you need to insert the data into your MongoDB?

if you need a near real time solution, then Kinesis and Lambdas is you best option (assuming you don't want to invest in 3rd party products). If you can afford a delay and do batching, then you can save the kinesis stream into S3 and then use AWS Glue to process and load your data into the database.

What you need to think is mostly what do you need to do with the data.

If you are collecting sensor data, where you only care about aggregations (eg clicks in a UI), then it is better if you store the raw data into s3 and then execute a data pipeline (using AWS Glue for example) to store the aggregated data into MongoDB. S3 will be faster and cheaper for those types of data.

If you are using the stream to pass business entities (eg documents that provide value on their own), then a near real time solution using AWS lambda will be a better choice.

Without knowing the exact use case, I would propose to store in your database only the data that provide value (eg reports on aggregated data) and use S3 with a lifecycle policy for the raw "sensor" data.