I am looking for all the methods for moving/copying the data from one folder to another on AWS S3 bucket.
Method 1: Via AWS CLI (Most easy)
Download and install awscli on ur instance, I am using here windows( 64-bit link ) and run "asw configure" to fill up your configuration and just run this single command on cmd
aws s3 cp s3://from-source/ s3://to-destination/ --recursive
Here cp for copy and recursive to copy all files
Method 2: Via AWS CLI using python
import os
import awscli
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
os.environ['LC_CTYPE'] = 'en_US.UTF-8'
from awscli.clidriver import create_clidriver
driver = create_clidriver()
driver.main('s3 mv s3://staging/AwsTesting/research/ s3://staging/AwsTesting/research_archive/ --recursive'.split())
Even this worked for me perfectly
Method 3: Via Boto using python
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'staging',
'Key': '/AwsTesting/research/'
}
s3.meta.client.copy(copy_source, 'staging', '/AwsTesting/research_archive/')
With my understanding I have assumed the 'key' for bucket is just the folder prefix so I have mentioned the folder path here
Error:
Invalid bucket name "s3://staging": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"
Even I changed it to simple bucket name as "staging" but no success.
How can I understand bucket connectivity via boto and the concept of this key?
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'staging',
'Key': 'AwsTesting/research/filename.csv'
}
s3.meta.client.copy(copy_source, 'staging', 'AwsTesting/')
An alternative to using cp
with the CLI is sync
- https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
aws s3 sync s3://mybucket s3://mybucket2
It will essentially do the same thing.
Use the following snippet which is working.
def s3candidateavtarcopy(old,new):
try:
response = s3.list_objects_v2(Bucket = s3_candidate_bucket,Prefix=old)
keycount = response['KeyCount']
if(keycount > 0):
for key in response['Contents']:
file = key['Key']
try:
output = file.split(old)
newfile = new + output[1]
input_source = {'Bucket': s3_candidate_bucket,'Key' : file }
s3_resource.Object(s3_candidate_bucket,newfile).copy_from(CopySource=input_source)
except ClientError as e:
print(e.response['Error']['Message'])
else:
print('Success')
else:
print('No matching records')
except ClientError as e:
print(e.response['Error']['Message'])
else:
print('Operatio completed')
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.