两个不同帐户/连接之间来自boto3或boto api的存储桶/密钥的并行副本

Question

I want to copy keys from buckets between 2 different accounts using boto3 api's. In boto3, I executed the following code and the copy worked

source =  boto3.client('s3')
destination = boto3.client('s3')
destination.put_object(source.get_object(Bucket='bucket', Key='key'))

Basically I am fetching data from GET and pasting that with PUT in another account.

On Similar lines in boto api, I have done the following

source = S3Connection()
source_bucket = source.get_bucket('bucket')
source_key = Key(source_bucket, key_name)

destination = S3Connection()
destination_bucket = destination.get_bucket('bucket')
dist_key = Key(destination_bucket, source_key.key)
dist_key.set_contents_from_string(source_key.get_contents_as_string())

The above code achieves the purpose of copying any type of data. But the speed is really very slow. I get around 15-20 seconds to copy data for 1GB. And I have to copy 100GB plus. I tried python mutithreading wherein each thread does the copy operation. The performance was bad as it took 30 seconds to copy 1GB. I suspect GIL might be the issue here. I did multiprocessing and I am getting the same result as of single process ie 15-20 seconds for 1GB file.

I am using a very high end server with 48 cores and 128GB RAM. The network speed in my environment is 10GBPS. Most of the search results tell about copying data between buckets in same account and not across accounts. Can anyone please guide me here. Is my approach wrong? Does anyone have a better solution?

Answer 1

Yes, it is wrong approach.

You shouldn't download the file. You are using AWS infrastructure, so you should make use of the efficient AWS backend call to do the works. Your approach is wasting resources.

boto3.client.copy will do the job better than this.

In addition, you didn't describe what you are trying to achieve(eg is this some sort of replication requirement? ).

Because with proper understanding of your own needs, it is possible that you don't even need a server to do the job : S3 Bucket events trigger, lambda etc can all execute the copying job without a server.

To copy file between two different AWS account, you can checkout this link Copy S3 object between AWS account

Note :

S3 is a huge virtual object store for everyone, that's why the bucket name MUST be unique. This also mean, the S3 "controller" can done a lot of fancy work similar to a file server , eg replication,copy, move file in the backend, without involving network traffics.

As long as you setup the proper IAM permission/policies for the destination bucket, object can move across bucket without additional server.

This is almost similar to file server. User can copy file to each other without "download/upload", instead, one just create a folder with write permission for all, file copy from another user is all done within the file server, with fastest raw disk I/O performance. You don't need powerful instance nor high performance network using backend S3 copy API.

Your method is similar to attempt FTP download file from user using the same file server, which create unwanted network traffics.

Answer 2

You should check out the TransferManager in boto3. It will automatically handle the threading of multipart uploads in an efficient way. See the docs for more detail.

Basically you must have to use the upload_file method and TransferManager will take care of the rest.

import boto3

# Get the service client
s3 = boto3.client('s3')

# Upload tmp.txt to bucket-name at key-name
s3.upload_file("tmp.txt", "bucket-name", "key-name")