Listing S3 Bucket newest file with Bash Script

Question

Hi am new to the world of shell scripting, and am Running into a couple of issues. I am writing a bash script that calls two AWS S3 Buckets, and lists the newest files that have been uploaded. On the 'long run', those two files have to be downloaded, and using CLI copied to 2 existing tables within an RDS PostgreSQL Database. So far, this is what I have:

#!/bin/bash


printf "Looking up newest files added...\n";

AUTO= $(aws s3 ls bucketx --recursive | sort | tail -n 1| awk '{print $4}');
IMMO= $(aws s3 ls buckety --recursive | sort | tail -n 1 | awk '{print $4}');

echo AUTO;
echo IMMO;

Problem is, and this is what I don't get, when I execute the AWS commands directly in the terminal, and goes well and behaves the way it should. But within the script, I get the following errors:

Looking up newest files added...
./shellTest.sh: line 6: bucketx/14.9.2016.csv: No such file or directory
./shellTest.sh: line 7: buckety/14.9.2016.csv: No such file or directory

Can someone tell me what I am doing wrong? Is it a logic-problem or 'just' a syntactical problem? I was expecting the 'echo' command to just print out the content of the variables x and y, but instead nothing is printed..

Cheers

Answer 1

what you want in AUTO is the value of the latest key so you need to have a string

AUTO="$(aws s3 ls bucketx --recursive | sort | tail -n 1| awk '{print $4}')"
IMMO="$(aws s3 ls buckety --recursive | sort | tail -n 1 | awk '{print $4}')"

so AUTO and IMMO will hold the value (the string value of the key) - if you omit the quote, the system will try to find the reference and will complain the file does not exist locally

To show the value you then need to have

echo $AUTO
echo $IMMO

see below

This one does not work

$ AUTO= $(aws s3 ls <bucket> --recursive | sort | tail -n 1| awk '{print $4}');
-bash: database/instantclient-sdk-linux.x64-11.2.0.4.0.zip: No such file or directory

but this one is ok

$ AUTO="$(aws s3 ls <bucket> --recursive | sort | tail -n 1| awk '{print $4}')"
$ echo $AUTO
database/instantclient-sdk-linux.x64-11.2.0.4.0.zip