How to move a File from One folder to Another Folder in the same AWS S3 bucket using Lambda?

Question

I am trying to Automate File movement from One folder to another folder within the same S3 bucket on the file creation event in the S3 bucket.

I was hoping to use Lambda function's triggers to do this but I feel, Lambda triggers at the Root directory level and can not use it at the Folder Level.

Example:

Bucket Name: my-only-s3-bucket 
Source Folder: s3://my-only-s3-bucket/Landing 
Target Folder: s3://my-only-s3-bucket/Staging

Requirement:

When a file gets created or uploaded into, Source Folder: s3://my-only-s3-bucket/Landing , it should get moved to s3://my-only-s3-bucket/Staging automatically without any manual intervention

How to achieve this?

Answer 1

I was hoping to use Lambda function's triggers to do this but I feel, Lambda triggers at the Root directory level and can not use it at the Folder Level.

This is not true. S3 has no concept of folders. You can trigger at any "level" using a filter prefix ie prefix -> "Landing/" and/or a suffix (as example ".jpg").

S3 trigger will call the lambda and pass the event with the new object as input. Then just use any language you are familiar with and use s3 copy built in function from any of the available AWS SDK (.Net, Java, python, etc..) to copy to the destination.

example:

def object_copied?(
  s3_client,
  source_bucket_name,
  source_key,
  target_bucket_name,
  target_key)

  return true if s3_client.copy_object(
    bucket: target_bucket_name,
    copy_source: source_bucket_name + '/' + source_key,
    key: target_key
  )
rescue StandardError => e
  puts "Error while copying object: #{e.message}"
end

Answer 2

I think the concept of relative path can solve your problem. Here's the code snippet that solves your problem using a library called s3pathlib , a objective-oriented s3 file system interface.

# import the library
from s3pathlib import S3Path

# define source and target folder
source_dir = S3Path("my-only-s3-bucket/Landing/")
target_dir = S3Path("my-only-s3-bucket/Staging/")

# let's say you have a new file in Landing folder, the s3 uri is
s3_uri = "s3://my-only-s3-bucket/Landing/my-subfolder/data.csv"

# I guess you want to cut the file to the new location and delete the original one
def move_file(p_file, p_source_dir, p_target_dir):
    # validate if p_file is inside of p_source_dir
    if p_file.uri.startswith(p_source_dir.uri):
        raise ValueError

    # find new s3 path based on the relative path
    p_file_new = S3Path(
        p_target_dir, p_file.relative_to(p_source_dir)
    )

    # move
    p_file.move_to(p_file_new)

    # if you want copy you can do p_file.copy_to(p_file_new)

# then let's do your work
if __name__ == "__main__":
    move_file(
        p_file=S3Path.from_s3_uri(s3_uri),
        p_source_dir=source_dir,
        p_target_dir=target_dir,
    )

If you want more advanced path manipulation, you can reference this document . And the S3Path.change(new_abspath, new_dirpath, new_dirname, new_basename, new_fname, new_ext) would be the most important one you need to know.