AWS CLI does not work in cron

Question

I'm trying to use the aws cli cp command in a cron of an aws environment on a Ubuntu 14.04.3 AWS EC2.

The ec2-user is called ubuntu and lives in /home/ubuntu

I have my aws config file in /home/ubuntu/.aws/config

[default]
output=json  
region=eu-central-1

I have my aws credentials file in /home/ubuntu/.aws/credentials

[default]
aws_access_key_id=******
aws_secret_access_key=******

My crontab looks like this

* * * * * sh /home/ubuntu/test.sh

The shell script tries to copy a test file over to S3 is a one-liner:

/usr/local/bin/aws s3 cp test.txt s3://<my-bucket>/test.txt >> /home/ubuntu/some-log-file.log

The cron runs the script each minute, but nothing is copied to the S3 bucket.

If i run the script manually on my shell it works.

I tried (without success):

Putting the right path in front of aws (/usr/local/bin/aws)

Putting aws_access_key_id and aws_secret_access_key into the .aws/config file as well.

Putting aws env vars to crontab and/or shell script

AWS_DEFAULT_REGION=eu-central-1
AWS_ACCESS_KEY_ID=******
AWS_SECRET_ACCESS_KEY =******

Defining HOME in the crontab and/or shell script

HOME="/home/ubuntu"

Putting the config and credential file location to the crontab

AWS_CONFIG_FILE="/home/ubuntu/.aws/config"
AWS_CREDENTIAL_FILE="/home/ubuntu/.aws/credentials"

Putting PATH to the crontab and/or the shell script

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:"

Has anybody an idea what I might do wrong?

Answer 1

Found out that I forgot an absolute path to test.txt (/home/ubuntu/test.txt)

I'll keep the question because it lists several options and might still be helpful to others.

Answer 2

Fix was relatively simple. When running AWS CLI commands from cron you need to set the user environment variables.

In the cron command use . $HOME/.profile;

Example:

10 5 * * * . $HOME/.profile;  /var/www/rds-scripts/clonedb.sh

In the shell script set the $SHELL and $PATH variables.

export SHELL=/bin/bash
export PATH=/usr/local/sbain:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

With these changes the AWS CLI is able to load the user credential files and locate the AWS CLI binary files.