AWS Lambda in Python to copy new files to another s3 bucket

Question

I have created a Lambda which is triggered every time I add a file to s3://test-bucket-01/, it copies all the files in s3://test-bucket-01/ to s3://test-bucket-02/

I would like it to only add the new file that has just been added?

Current Code:

import boto3
s3 = boto3.resource('s3')


def lambda_handler(event, context):
    bucket = s3.Bucket('test-bucket-01')
    dest_bucket = s3.Bucket('tb-bucket-02')
    print(bucket)
    print(dest_bucket)

    for obj in bucket.objects.all():
        dest_key = obj.key
        print(dest_key)
        s3.Object(dest_bucket.name, dest_key).copy_from(CopySource = {'Bucket': obj.bucket_name, 'Key': obj.key})

Results:

Function Logs:
START RequestId: XXXXXXX-XXXXXXX-XXXXXXX Version: $LATEST
s3.Bucket(name='test-bucket-01')
s3.Bucket(name='test-bucket-02')
test-data-01.json
test-data-02.json
test-data-03.json

Answer 1

You not need to list the bucket to copy all files because the event variable is a json that contains the file that you has been copied into test-bucket-01.

Example:

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventTime": "1970-01-01T00:00:00.000Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "EXAMPLE"
      },
      "requestParameters": {
        "sourceIPAddress": "127.0.0.1"
      },
      "responseElements": {
        "x-amz-request-id": "C3D13FE58DE4C810",
        "x-amz-id-2": "FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "testConfigRule",
        "bucket": {
          "name": "sourcebucket",
          "ownerIdentity": {
            "principalId": "EXAMPLE"
          },
          "arn": "arn:aws:s3:::mybucket"
        },
        "object": {
          "key": "key/to/HappyFace.jpg",
          "size": 1024,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e"
        }
      }
    }
  ]
}

In this case, this event notify ObjectCreated:Put event. A new file has been copied to "sourcebucket" bucket with key "key/to/HappyFace.jpg".

Answer 2

Why don't you want to use native functional of AWS S3 to replicate new files from one bucket to another?

It's easier and more reliable. And you can replicate files by prefixes, change storage plans and many others with this native feature.

Answer 3

got the same problem.

How do you fix the issue of multiple times copy of the same file?

Answer 4

so what is the answer ? how to trigger the lambda only when a new file is added?