Bash script works fine in terminal but same script run in Chef fails

Question

I'm running Chef-client 11.12.8. I've written a bash script which contains the following line:

aws s3 ls s3://my-bucket/prod-builds/

I'm just using the awscli tools to get a list of files from an S3 bucket. When I run this manually on my terminal (Ubuntu 14.04 LTS), I get this output:

2014-10-10 21:27:50   42081271 aragorn-38d5afe23f1fa09a991c4cdbcb8461ed8709d15f.tgz

But when chef-client executes a bash script with this same line, using printf statements, I see that the bash script outputs only:

2014-10-10

It seems that the presence of spaces in the command output is confusing chef-client.

Here's the line from my chef recipe that executes the bash script:

execute "Download latest build" do
    command "bash /path/to/my/script"
    creates "/file/that/gets/created"
    user "aragorn"
    action :run
end

Any idea how I can fix this?

Update: As requested, here's a portion of my script:

if [ -z "$gitCommitId" ]; then
        # Take a filename like "aragorn-38d5afe23f1fa09a991c4cdbcb8461ed8709d15f.tgz" and extract the commitId from it
        # Note that we take the topmost result
        gitCommitId=$(aws s3 ls s3://my-bucket/prod-builds/ | sort -r | head -2 | sed -n '1!p' | awk -F "aragorn-" '{print $2}' | cut -d'.' -f 1)
fi

I've confirmed that this line executes (using printf statements that output), but the strange thing is that when I execute the command aws s3 ls s3://my-bucket/prod-builds/ | sort -r | head -2 | sed -n '1!p' | awk -F "aragorn-" '{print $2}' | cut -d'.' -f 1 aws s3 ls s3://my-bucket/prod-builds/ | sort -r | head -2 | sed -n '1!p' | awk -F "aragorn-" '{print $2}' | cut -d'.' -f 1 aws s3 ls s3://my-bucket/prod-builds/ | sort -r | head -2 | sed -n '1!p' | awk -F "aragorn-" '{print $2}' | cut -d'.' -f 1 directly in my terminal (or manually run the script from my terminal), it works fine. But when run from chef the gitCommitId gets set to nothing.

Update #2: I wound up writing a simpler bash script to test differences between Chef and Terminal. I confirmed that the Chef run outputs different results from calling aws s3 ls s3://my-bucket/prod-builds than when calling it from the terminal. Specifically, it left out a folder in the S3 bucket that was present in the terminal call. I still can't explain this.

I wound up using the following line which gave me consistent results by ensuring that only builds were displayed, and no folders:

aws s3 ls s3://my-bucket/prod-builds/ | grep aragorn | sort -r | head -1 | cut -d'-' -f 4 | cut -d'.' -f 1

Special thanks to both @TejayCardon and @StephenC for pointing me in the right direction!

Answer 1

The problem is almost certainly occuring in the script, so you need to figure out why the script is failing.

I'd start by changing the command attribute to

    command "bash -x /path/to/my/script > /tmp/some_file 2>&1"

to see exactly what commands that bash is executing.

---

It seems that the presence of spaces in the command output is confusing chef-client.

Nothing you have shown us in the Question supports that conclusion. (And it is unlikely. The execute resource won't do anything with the output, except (maybe) log it.)

I suspect that the problem is with the script, or one of the commands it runs .... or the assumptions that it is making about environment variables, current directory, etc.

---

UPDATE

So the problem seems to be in this pipeline:

   aws s3 ls s3://my-bucket/prod-builds/ | \
       sort -r | \ 
       head -2 | \
       sed -n '1!p' | \
       awk -F "aragorn-" '{print $2}' | \
       cut -d'.' -f 1

I'd start by looking to see if the first command in the pipeline is producing any output onto stdout. (I suspect that the problem is that the environment variables that the aws command needs have not been set.)