Cloudwatch Log event aggregation pattern?

Question

I have a series of webservices and platform applications whose logs contain data about events pertinent to disparate parts of the overall platform. All of these logs are logged to Cloudwatch and my company is exclusively an AWS shop for new applications.

I would like to aggregate these log events in some sort of data store I can use to gain insights across the platform and conduct some predictive analytics based on the event data.

What would be my best options for designing this? My initial thought would be to Firehose the data into a Dynamo table but I would love some input from those more familiar with this type of problem.

Answer 1

First of all, Did you try CW Log Insight ?

Then, I believe you are much better off streaming your CW logs to ElasticSearch. There is native support for it and as ES is a pretty standard platform for log-analytics, chances are you will get better community support and better tool. Really save you the headaches you can get into trying to implement something new for DynamoDB