EC2 instance running S3 Sync command terminates before data transfer is complete

Question

I have an EC2 instance running Linux. This instance is used to run aws s3 commands.

I want to sync the last 6 months worth of data from source to target S3 buckets. I am using credentials with the necessary permissions to do this.

Initially I just ran the command:

aws s3 sync "s3://source" "s3://target" --query "Contents[?LastModified>='2022-08-11' && LastModified<='2023-01-11']"

However, after maybe 10 mins this command stops running, and only a fraction of the data is synced.

I thought this was because my SSM session was terminating, and with it the command stopped executing.

To combat this, I used the following command to try and ensure that this command would continue to execute even after my SSM terminal session was closed:

nohup aws s3 sync "s3://source" "s3://target" --query "Contents[?LastModified>='2022-08-11' && LastModified<='2023-01-11']" --exclude "*.log" --exclude "*.bak" &

Checking the status of the EC2 instance, the command appears to run for about 20 mins, before clearly stopping for some reason.

Answer 1

The --query parameter controls what information is displayed in the response from an API call.

It does not control which files are copied in an aws s3 sync command. The documentation for aws s3 sync defines the --query parameter as: "A JMESPath query to use in filtering the response data."

Your aws s3 sync command will be synchronizing ALL files unless you use Exclude and Include Filters . These filters operate on the name of the object. It is not possible to limit the sync command by supplying date ranges.

I cannot comment on why the command would stop running before it is complete. I suggest you redirect output to a log file and then review the log file for any clues.