How to copy an entire “folder” to another path using S3 with sdk?

Question

When I do for a single file it works:

    aws_s3 = AWS::S3.new(S3_CONFIG)
    bucket = aws_s3.buckets[S3_CONFIG["bucket"]]

    object = bucket.objects["user/1/photos/image_1.jpg"]
    new_object = bucket.objects["users/1/photos/image_1.jpg"]
    object.copy_to new_object, {:acl => :public_read}

But I want to move the entire "/photos" folder throws No Such Key . Probably the s3 keys are only the full path for each file. How to do that?

    aws_s3 = AWS::S3.new(S3_CONFIG)
    bucket = aws_s3.buckets[S3_CONFIG["bucket"]]

    object = bucket.objects["user/1/photos"]
    new_object = bucket.objects["users/1/photos"]
    object.copy_to new_object, {:acl => :public_read}

Thanks!

Answer 1

Did it:

bucket.objects.with_prefix("user/1/photos").each do |object|
   ...
end

Answer 2

I needed additional code to get this working. Basically chop off the base from the source prefix, then add that to the destination prefix:

def copy_files_s3(bucket_name, source, destination)
  source_bucket = @s3.buckets[bucket_name]
  source_bucket.objects.with_prefix(source).each do |source_object|
    new_file_name = source_object.key.dup
    new_file_name.slice! source
    new_object = source_bucket.objects["#{destination}#{new_file_name}"]
    source_object.copy_to new_object, {acl: :public_read}
  end
end

Answer 3

A "folder" is not an object in S3, that is why you can not get it by key, but the folder path is actually a prefix for all the keys of the objects contained by the folder.

Another important thing, you have to url encode the keys otherwise you may end up with an unknown key error.

require 'aws-sdk'
require 'aws-sdk-s3'
require 'securerandom'
require 'uri'
require "erb"
include ERB::Util

def copy_folder(folder, destination)
    bucket_name = 'your_bucket'
    credentials = Aws::Credentials.new('key', 'secret')
    s3_client = Aws::S3::Client.new(region:'the_region', credentials: credentials)
    enumerate_keys_with_prefix(source).each do |source_object|
       source_key = url_encode(source_object.key)
       destination_key = source_object.key.dup.sub(source, "")
       s3_client.copy_object({bucket: bucket_name, copy_source: bucket_name+'/'+source_key, key: destination+'/'+destination_key, acl: "public-read"})
    end
end

def enumerate_keys_with_prefix(prefix)
    bucket_name = 'your_bucket'
    credentials = Aws::Credentials.new('key', 'secret')
    s3 = Aws::S3::Resource.new(region:'the_region', credentials:credentials)
    return s3.bucket(bucket_name).objects(prefix: prefix)
end