AWS CloudWatch | 将日志导出到EC2服务器

Question

I have "cloudwatch" service to monitor logs for my EC2 running instances. But the ColudWatch web console does not seem to have a button to allow you to download/exporting the log data from it.

Any ideas how I can achieve this goal through CLI or GUI?

Answer 1

Programmatically, using boto3 (Python),

log_client=boto3.client('logs')
result_1=log_client.describe_log_streams(logGroupName='<NAME>')

(I don't know what log group names for EC2 instances look like; for Lambda they are of the form '/aws/lambda/FuncName' . Try grabbing the names you see in the console).

result_1 contains two useful keys: logStreams (the result you want) and nextToken (for pagination, I'll let you look up the usage).

Now result_1['logStreams'] is a list of objects containing a logStreamName . Also useful are firstEventTimestamp and lastEventTimestamp .

Now that you have log stream names, you can use

log_client.get_log_events(logGroupName='<name>',logStreamName='<name>'

The response contains nextForwardToken and nextBackwardToken for pagination, and events for the log events you want. Each event contains a timestamp and a message .

I'll leave it to you to look up the API to see what other parameters might be useful to you. By the way, the console will let you stream your logs to an S3 bucket or to AWS's ElasticSearch service. ElasticSearch is a joy to use, and Kibana's UI is intuitive enough that you can get results even without learning their query language.

Answer 2

You can use the console or the AWS CLI to download CloudWatch logs to Amazon S3. You do need to know the log group name, from & to timestamps in the log, destination bucket and prefix. Amazon recommends a separate S3 bucket for your logs. Once you have a bucket you create an export task, under (in the console) Navigation - Logs - select your log group - Actions - Export data to S3 - fill in the details for your export - select Export data. Amazon's documentation explains it pretty well at: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html . And CLI instructions are there also if you wanted to use that. I imagine with the CLI you could also script your export, but you would have to define the variables somehow so you don't overwrite an existing export.

If this is part of your overall AWS disaster recovery planning, you might want to check out some tips & best practices, such as Amazon's white paper on AWS disaster recovery, and NetApp's discussion of using the cloud for disaster recovery .