将文件从S3中复制到Ubuntu 18.04中的临时文件夹的计划过程

Question

Looking for recommendations for the following scenario:

In an ubuntu 18.04 server, every 1 minute check for new files in an AWS S3 bucket, fetch only the newest file to a temp folder at the end of the day remove them.

It should be automated in bash.

I proposed using aws s3 events notification, queues, lambda but it was defined that is best to keep it simple.

i am looking for recommendations for the steps described below:

For step 1 i was doing aws s3 ls | awk (FUNCTION to filter files updated within the last minute) aws s3 ls | awk (FUNCTION to filter files updated within the last minute) then i realized that it was best to do it with grep

 0-Cron job should run from 7:00 to 23:00 every minute 1-List the files updated to S3 bucket during the past 1 minute 2-List the files in a temp-encrypted folder in ubuntu 18.03 3-Are the files listed in step 1 already downloaded in folder temp-encrypted from step 2 4-If the files are not already donloaded > download newest files from S3 bucket into temp-encrypted 5-At end of the day 23:00 take a record of the last files fetched from s3 6-run cleanup script at end of the day to remove everything in temp-encrypted 

I attach a diagram with the intended process and infrastructure design.

Answer 1

The solution was like this:

1. Change FTPS to SFTP running in Ubuntu 18.04
2. change main ports: randomport1 for SSH and randomport2 for SFTP
3. configure SFTP in sshd_config file
4. once everything is working create local directory structure
5. by using a bash script

5.1 List what is in S3 and save in a var

5.2 for each of the files listed in s3 check if there is a new file not present in the mirrored file in the local directory s3-mirror

5.3 if there is new file fetch, touch a file with empy contents in s3-mirror directory just same name, move encrypted file to SFTP and remove fetched S3 file from mirrored local directory

5.4 record successful actions in a log.

So far it works good.