导出任务运行时会创建一个新的 object：将 cloudwatch 日志导出到 s3 存储桶

Question

I'm trying to move CloudWatch logs from a log groups to an s3 bucket

in the bucket I have three objects (folders): Test, Dev, Prod

When I create an export task specifying the dev folder for example as an S3 bucket prefix, the logs stream are successfully exported, but the problem is another folder is created with the name of the taskID in dev folder as a subfolder !

Any way to put the logs stream directly to the dev folder or this is uncontrolled behavior?

Answer 1

There is no way to send the logs as files only using export task , as from AWS point of view you export a collection of items. Another problem, is that export task cannot be done automatically, it's just a manual step (Not suitable if you want to do this operation multiple times for the whole cloud watch logs, or whenever there is a new logs generated.)

Another Solution:

You could implement your own AWS lambda to be triggered based on scheduled time to import only the files from every AWS cloudWtach log Group you want.

Also, storing logs can get bigger and bigger in the future and you don't always need to retrieve them in a short period, so it's not a great solution for storing logs on the long run from a Cost perspective, you could use Glacier for this purpose after you transfer your logs to AWS S3 using storage class transition

Corner Case: AWS only allows one Export Task running per account. This means that if the lambda function try to export multiple log groups at once, you will get a LimitExceededException error.

Please refer to this tutorial of how to implement such a solution.

Based on your comment on the answer:

No there is no way around for you to directly copy the logs from the logGroup as files, as by default AWS put the task_id of the exported logs as the name of the collection of the logs and this cannot be changed as it's uncontrolled behavior .

Also, There will be always a sub-directory that will be generated due to you need to set the destinationPrefix and if you didn't specify it in the request, then the default will be exportedlogs , as follow using Boto3 CloudWatchLogs :

response = logs.create_export_task(
            logGroupName=log_group_name,
            fromTime=from_to_time,
            to=export_to_time,
            destination=S3_BUCKET,
            destinationPrefix=custom_task_id.strip("/")
        )
        print("Task created: %s" % response['taskId'])

But you can Copy the log files after the CreateExportTask API finishes from the created sub-directory and then delete every file you copy, but before that you need to list all the items before every deletation to know when to stop and when to delete the sub-directory.

To summerize:

1- List items inside the bucket sub-directory.

2- copy a file to a new one (removing the taskId from the file name and leave the destinationPerfix ).

3- delete the original.

4- iterate until you copy and delete all the logs inside the taskId directory on a single run of your lambda OR you can invoke your lambda multiple times based on the number of the logs inside your taskId sub-directory .

4- Delete the whole sub-directory ( taskId directory)

This is a sample code, using boto3:

import sys, traceback, json
import boto3
s3 = boto3.client('s3')
BUCKET_NAME = '<Bucket Name>'
LOG_GROUP_NAME = 'export-task-test'


def copy_object(src_key, dst_key):
   try:
     result = s3.copy_object(
        Bucket=BUCKET_NAME,
        CopySource='%s/%s' % (BUCKET_NAME, src_key),
        Key=dst_key
    )
     result = True
   except:
     traceback.print_exc()
     result = False
return result


def delete_object(src_key):
  try:
    s3.delete_object(
        Bucket=BUCKET_NAME,
        Key=src_key
    )
    result = True
  except:
    traceback.print_exc()
    result = False
return result


def cleanup_objects(src_key):
   # Delete Old Object & aws-logs-write-test Object
   return delete_object(src_key) and delete_object('exportedlogs/aws-logs-write-test')


def move_log(src_key):
  result = False
  group_stream_log_list = [LOG_GROUP_NAME] + src_key.split('/')[-2:]
  # <LogGrpupName>/<LogStreamName>/<LogName>
  dst_key = '/'.join(group_stream_log_list)
  return copy_object(src_key, dst_key)


def move_log_and_cleanup_objects(record):
  src_key = record['s3']['object']['key']
  event_name_category = record['eventName'].split(':')[0]
  if event_name_category != 'ObjectCreated':
    return 'Skipped: %s %s' % (event_name_category, src_key)
  if not move_log(src_key):
    return 'Failed to move %s' % src_key
  if not cleanup_objects(src_key):
    return 'Failed to cleanup %s' % src_key
  return 'Successfully moved %s' % src_key


def lambda_handler(event, context):
  record = event['Records'][0]
  result = move_log_and_cleanup_objects(record)
  print(result)
  return result

Please, refer to this detailed solution of how to implement the above using lambda if your logs is not that intensive

This may be an intensive task based on the number of logs you have and also if you need to change the name of the TaskId for the whole logs you imported to your S3 bucket at once to cut-off the time needed to interrupt every log to change it's name, if you have thousands of logs, as this will add an overhead time, so you may need to go to another serverless architecture as using AWS ECS and AWS Fargate by putting all your code in a container and trigger based on a ECS Scheduled Task , which will start the Fargate container at certain times of day, the container will stop once the process exits, but you will achieve un-limited timeout this solution gives you and the best Cost you can get for this task.