upload custom logs in s3 to cloudwatch for metrics monitoring

Question

I created a custom app that automatically uploads logs to s3. Is there a way to push those logs to cloudwatch from s3 for analysis and alerting? I'm aware that I can use a cloudwatch agent to push directly to cloudwatch from the app but there are complications involved in that option. Thank you!

Answer 1

You could probably use Cloudwatch Events to listen to S3 changes. Not sure about if you can get the data from the S3 file, or just a trigger saying that a new log has been added.

You could also use S3 event notifications ( https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html ) connected either to a lambda or SQS, and from there create the logs to Cloudwatch. (similar to what was suggested by @marcin)

A better solution, but one that is a bit beyond the scope of the question, would be to send your logs through Kinesis Firehose and from there add the Cloudwatch and S3 logs.

Answer 2

I'm not aware of any out-of-the-box mechanism for that provided by AWS. But I think it could be relatively easy to develop .

Namely, you can create S3 notification for a PUT of a new log file from your app to S3. The event would trigger a lambda function . The function would get the file and using AWS SDK , eg boto3's put_log_events , it would send the log events to CloudWatch logs.