AWS CloudWatch Logs to ELK

Question

I am planning to copy the AWS CloudWatch Logs to ELK and want to use Kibana Dashboard to visualise the logs.

One option is to stream the logs from CloudWatch to ELK.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_ES_Stream.html

But I feel this will involve execution of Lambda functions extensively and it might not be a cost effective option.

Is there any other cost effective way to copy logs from CloudWatch to maybe S3 and then to ELK?

I am Ok, if the logs are not realtime, maybe a delay of 15 mins or maybe one hour is OK.

But I am looking for a cost effective solution.

Btw, what is the best way to purge the CloudWatch logs periodically? ( maybe after one week)

Answer 1

Firstly, you can set the retention policy for your cloudwatch logs. By default it's indefinite, but you can set it to say like 7 days as you mentioned.

Secondly, according to me lambda would be a cost effective option to ingest the logs to your elasticsearch domain.

Other option, l can think as below:

• ec2 instance with log stash reading the logs from cloudwatch and ingesting it to elasticsearch.
• probably you can extend above with aws batch

Answer 2

My answer is not exactly the answer to your case, it's an attempt to solve this problem from another angle.
It will be actual for you, if you own logs generation.

Copying logs from one source to another assumes more or less the same solution. But the whole idea makes me curious if you need CloudWatch at all (ie logs are handled by Elastic stack in the end).
In this case we don't need CloudWatch and your apps can send logs directly to Elastic stack (via Logstash) you have.
We need just to replace CloudWatch with Logstash in your logging configuration in apps. Some AWS services may not support it of course.

Answer 3

EC2 spot instance will be the cheapest option.
If you have few log entries (very unusual case), lambda will be cheaper. But this situation happens only in the beginning, there are usually tons of logs and using lambda for this would be costly :-)

EC2 spot instance can be interrupted any time but it's fine for us. Because we don't need realtime synchronization of logs.

You can play with EC2 Saving Plan costs, if you're going to support logs synchronization for 1 year or more.