I am trying to get the following bash script to work (copied from http://curl.haxx.se/mail/archive-2014-10/0006.html#replies ):
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET
${contentType}
${dateValue}
${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
I am getting a SignatureDoesNotMatch error no matter what I do.
Any ideas on how to fix this will be greatly appreciated.
After way too much time spent on this I finally got it to work:
This line:
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
is missing an 'e':
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
In other words, characters weren't being escaped before the string was signed.
As an aside, I also learned that for get requests, the content type is meaningless.
Using various answers in this thread, I converted it into a handy s3get
bash function:
#!/bin/bash
#usage - s3get writes the specified object to stdout
# s3get <bucket/key> [region]
#set these in your environment/profile (NOT HERE)
AWS_ACCESS_KEY=""
AWS_SECRET_KEY=""
#example usage
s3get my-bucket/a/path/to/my/file > /tmp/file
function s3get {
#helper functions
function fail { echo "$1" > /dev/stderr; exit 1; }
#dependency check
if ! hash openssl 2>/dev/null; then fail "openssl not installed"; fi
if ! hash curl 2>/dev/null; then fail "curl not installed"; fi
#params
path="${1}"
bucket=$(cut -d '/' -f 1 <<< "$path")
key=$(cut -d '/' -f 2- <<< "$path")
region="${2:-us-west-1}"
#load creds
access="$AWS_ACCESS_KEY"
secret="$AWS_SECRET_KEY"
#validate
if [[ "$bucket" = "" ]]; then fail "missing bucket (arg 1)"; fi;
if [[ "$key" = "" ]]; then fail "missing key (arg 1)"; fi;
if [[ "$region" = "" ]]; then fail "missing region (arg 2)"; fi;
if [[ "$access" = "" ]]; then fail "missing AWS_ACCESS_KEY (env var)"; fi;
if [[ "$secret" = "" ]]; then fail "missing AWS_SECRET_KEY (env var)"; fi;
#compute signature
contentType="text/html; charset=UTF-8"
date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
resource="/${bucket}/${key}"
string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${resource}"
signature=`echo -en $string | openssl sha1 -hmac "${secret}" -binary | base64`
#get!
curl -H "x-amz-date: ${date}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${access}:${signature}" \
"https://s3-${region}.amazonaws.com${resource}"
}
Tested on OSX and Ubuntu. Saved in this Github gist .
The TS asked for a working SHA-1 version of the script. However, SHA-1 is outdated and Amazon has datacenters that only accept SHA-256 encryption, hereby the download script that can be used for all S3 datacenters: It also follows HTTP 307 redirects.
#!/bin/sh
#USAGE:
# download-aws.sh <bucket> <region> <source-file> <dest-file>
set -e
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
file=$3
bucket=$1
host="${bucket}.s3.amazonaws.com"
resource="/${file}"
contentType="text/plain"
dateValue="`date +'%Y%m%d'`"
X_amz_date="`date +'%Y%m%dT%H%M%SZ'`"
X_amz_algorithm="AWS4-HMAC-SHA256"
awsRegion=$2
awsService="s3"
X_amz_credential="$s3Key%2F$dateValue%2F$awsRegion%2F$awsService%2Faws4_request"
X_amz_credential_auth="$s3Key/$dateValue/$awsRegion/$awsService/aws4_request"
signedHeaders="host;x-amz-algorithm;x-amz-content-sha256;x-amz-credential;x-amz-date"
contentHash="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
HMAC_SHA256_asckey () {
var=`/bin/echo -en $2 | openssl sha256 -hmac $1 -binary | xxd -p -c256`
echo $var
}
HMAC_SHA256 () {
var=`/bin/echo -en $2 | openssl dgst -sha256 -mac HMAC -macopt hexkey:$1 -binary | xxd -p -c256`
echo $var
}
REQUEST () {
canonicalRequest="GET\n$resource\n\n"\
"host:$1\n"\
"x-amz-algorithm:$X_amz_algorithm""\n"\
"x-amz-content-sha256:$contentHash""\n"\
"x-amz-credential:$X_amz_credential""\n"\
"x-amz-date:$X_amz_date""\n\n"\
"$signedHeaders\n"\
"$contentHash"
#echo $canonicalRequest
canonicalHash=`/bin/echo -en "$canonicalRequest" | openssl sha256 -binary | xxd -p -c256`
stringToSign="$X_amz_algorithm\n$X_amz_date\n$dateValue/$awsRegion/s3/aws4_request\n$canonicalHash"
#echo $stringToSign
s1=`HMAC_SHA256_asckey "AWS4""$s3Secret" $dateValue`
s2=`HMAC_SHA256 "$s1" "$awsRegion"`
s3=`HMAC_SHA256 "$s2" "$awsService"`
signingKey=`HMAC_SHA256 "$s3" "aws4_request"`
signature=`/bin/echo -en $stringToSign | openssl dgst -sha256 -mac HMAC -macopt hexkey:$signingKey -binary | xxd -p -c256`
#echo signature
authorization="$X_amz_algorithm Credential=$X_amz_credential_auth,SignedHeaders=$signedHeaders,Signature=$signature"
result=$(curl --silent -H "Host: $1" -H "X-Amz-Algorithm: $X_amz_algorithm" -H "X-Amz-Content-Sha256: $contentHash" -H "X-Amz-Credential: $X_amz_credential" -H "X-Amz-Date: $X_amz_date" -H "Authorization: $authorization" https://${1}/${file} -o "$2" --write-out "%{http_code}")
if [ $result -eq 307 ]; then
redirecthost=`cat $2 | sed -n 's:.*<Endpoint>\(.*\)</Endpoint>.*:\1:p'`
REQUEST "$redirecthost" "$2"
fi
}
REQUEST "$host" "$4"
Tested on Ubuntu
If someone knows a solution to remove the HMAC-ASCII step, you're welcome to reply. I got this only working in this way.
bucket=your-bucket-name
contentType="text/plain"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxx
s3Secret=xxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
file1=file-name
resource1="/${bucket}/${file1}"
curl -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" "https://s3-us-west-2.amazonaws.com/${resource1}" -o "file-name-to-save-the-output"
I was getting errors in the actual answer. This works for me. This will get the file as what it is and not as string.
It required minor adjustment, but the following lines works well
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" https://${bucket}.s3.amazonaws.com/${file}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.