I am new to AWS CLI and have requirement to upload files to S3 from a Linux server. Below is my use-case:
Input directory has structure where folder is created on server based on the current date like: /users/data/20200329
and /users/data/20200330
.
Each folder contains multiple sub folders:
/users/data/20200329/govt/inbound/truey.gz
/users/data/20200329/corp/inbound/672695.gz
/users/data/20200330/govt/inbound/dddd.gz
/users/data/20200330/corp/inbound/4444.gz
I want to upload to S3 like:
s3://bucketname/20200329/truey.gz
s3://bucketname/20200329/672695.gz
s3://bucketname/20200330/dddd.gz
s3://bucketname/20200330/4444.gz
and likewise.
So in nutshell I don't want subfolders in S3 but just single date folder which would contain all files from sub folders mentioned in Input directory. How do I achieve this with the AWS CLI?
The logic for putting files into specific paths is beyond what the AWS CLI can determine .
You would need to issue individual commands:
aws s3 cp /users/data/20200329/govt/inbound/truey.gz s3://bucketname/20200329/truey.gz
aws s3 cp /users/data/20200329/corp/inbound/672695.gz s3://bucketname/20200329/672695.gz
aws s3 cp /users/data/20200330/govt/inbound/dddd.gz s3://bucketname/20200330/dddd.gz
aws s3 cp /users/data/20200330/corp/inbound/4444.gz s3://bucketname/20200330/4444.gz
If you wanted to apply some form of logic for where files should be uploaded, I would recommend writing a script (eg in Python) that will:
upload_file()
API call to upload the file to the desired locationThis would deal with all the files for a single day:
aws s3 cp /users/data/20200329/govt/inbound/ \
s3://bucketname/20200329 \
--recursive
aws s3 cp /users/data/20200329/corp/inbound/ \
s3://bucketname/20200329 \
--recursive
If you want to iterate over days of the months and so on:
#!/bin/bash
#requires Bash 4.0+ to pad leading 0 in day number
for day in {01..31};
do
aws s3 cp /users/data/20200329/govt/inbound/ \
s3://bucketname/202003$day \
--recursive
aws s3 cp /users/data/20200329/corp/inbound/ \
s3://bucketname/202003$day \
--recursive
done
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.