從 Amazon S3 存儲桶下載文件的腳本

Question

嘗試編寫腳本以從 Amazon S3 存儲桶下載文件。

cURL 站點上的示例有問題。 下面的腳本產生：

我們計算的請求簽名與您提供的簽名不匹配。 檢查您的密鑰和簽名方法。

感謝任何幫助。

#!/bin/sh 
file="filename.php"
bucket="my-bucket"
resource="/${bucket}/${file}"
contentType="text/html"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key='ABCABCABCABCABCABCAB'
s3Secret='xyzxyzyxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzx'
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -      binary | base64`

curl -v -H "Host:lssngen-updates-east.s3.amazonaws.com" \
        -H "Date:${dateValue}" \
        -H "Content-Type:${contentType}" \
        -H "Authorization: AWS ${s3Key}:${signature}" \
        https://${bucket}.s3.amazonaws.com/${file}

Answer 1

避免自己簽署請求，很多事情可能會出錯或很難做到。 例如，您應該檢查日期是否設置為 GMT 或使用 x-amz-date 標頭。

另一種方法是使用AWS 命令​​行界面，因此使用$ aws s3 cp或$ aws s3 sync 。

Answer 2

我寫這個 bash 腳本是為了從 s3 下載文件（我下載的是壓縮文件，你可以更改 contentType 來下載其他類型的文件）

#!/bin/sh 
outputFile="Your_PATH"
amzFile="AMAZON_FILE_PATH"
bucket="YOUR_BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="YOUR_S3_KEY"
s3Secret="YOUR_S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`

curl  -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile

Answer 3

截至 2019 年 8 月，我發現此方法有效。 增加了區域，URL 的格式也發生了變化。

#!/bin/sh 
outputFile="/PATH/TO/LOCALLY/SAVED/FILE"
amzFile="BUCKETPATH/TO/FILE"
region="YOUR-REGION"
bucket="SOME-BUCKET"
resource="/${bucket}/${amzFile}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave our "TZ=GMT" if your system is already GMT (but don't have to)
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: s3-${region}.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://s3-${region}.amazonaws.com/${bucket}/${amzFile} -o $outputFile

Answer 4

#!/bin/sh
# This works for cross region
outputFile="/PATH/TO/FILE"
awsFile="BUCKETPATH/TO/FILE"
bucket="SOME-BUCKET"
resource="/${bucket}/${awsFile}"
contentType="application/x-compressed-tar"
# Change the content type as desired
dateValue=`TZ=GMT date -R`
#Use dateValue=`date -R` if your TZ is already GMT
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -n ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${awsFile} -o $outputFile

Answer 5

我創建了一個完整的 AWS 簽名創建者。 你可以在這里找到原始文件（請注意：它有多個要點注釋，所以它絕對有效！）。

readonly AWS_ACCESS_KEY_ID='<your_access_key_id>'
readonly AWS_SECRET_ACCESS_KEY='<your_secret_access_key>'
readonly AWS_SERVICE='s3'
readonly AWS_REGION='us-east-1'
readonly AWS_S3_BUCKET_NAME='<your_bucket_name>'
readonly AWS_SERVICE_ENDPOINT_URL="\
${AWS_S3_BUCKET_NAME}.${AWS_SERVICE}.amazonaws.com"

# Create an SHA-256 hash in hexadecimal.
# Usage:
#   hash_sha256 <string>
function hash_sha256 {
  printf "${1}" | openssl dgst -sha256 | sed 's/^.* //'
}

# Create an SHA-256 hmac in hexadecimal format.
# Usage:
#   hmac_sha256 <key> <data>
function hmac_sha256 {
  key="$1"
  data="$2"
  printf "${data}" | openssl dgst -sha256 -mac HMAC -macopt "${key}" | \
      sed 's/^.* //'
}

readonly CURRENT_DATE_DAY="$(date -u '+%Y%m%d')"
readonly CURRENT_DATE_TIME="$(date -u '+%H%M%S')"
readonly CURRENT_DATE_ISO8601="${CURRENT_DATE_DAY}T${CURRENT_DATE_TIME}Z"

readonly HTTP_REQUEST_METHOD='GET'
readonly HTTP_REQUEST_PAYLOAD=''
readonly HTTP_REQUEST_PAYLOAD_HASH="$(printf "${HTTP_REQUEST_PAYLOAD}" | \
    openssl dgst -sha256 | sed 's/^.* //')"
readonly HTTP_CANONICAL_REQUEST_URI='/video_clips/0940.m3u8'
readonly HTTP_CANONICAL_REQUEST_QUERY_STRING=''
readonly HTTP_REQUEST_CONTENT_TYPE='application/x-www-form-urlencoded'

readonly HTTP_CANONICAL_REQUEST_HEADERS="\
content-type:${HTTP_REQUEST_CONTENT_TYPE}
host:${AWS_SERVICE_ENDPOINT_URL}
x-amz-content-sha256:${HTTP_REQUEST_PAYLOAD_HASH}
x-amz-date:${CURRENT_DATE_ISO8601}"
# Note: The signed headers must match the canonical request headers.
readonly HTTP_REQUEST_SIGNED_HEADERS="\
content-type;host;x-amz-content-sha256;x-amz-date"

readonly HTTP_CANONICAL_REQUEST="\
${HTTP_REQUEST_METHOD}
${HTTP_CANONICAL_REQUEST_URI}
${HTTP_CANONICAL_REQUEST_QUERY_STRING}
${HTTP_CANONICAL_REQUEST_HEADERS}\n
${HTTP_REQUEST_SIGNED_HEADERS}
${HTTP_REQUEST_PAYLOAD_HASH}"

# Create the signature.
# Usage:
#   create_signature
function create_signature {
  stringToSign="AWS4-HMAC-SHA256
${CURRENT_DATE_ISO8601}
${CURRENT_DATE_DAY}/${AWS_REGION}/${AWS_SERVICE}/aws4_request
$(hash_sha256 "${HTTP_CANONICAL_REQUEST}")"

  dateKey=$(hmac_sha256 key:"AWS4${AWS_SECRET_ACCESS_KEY}" \
      "${CURRENT_DATE_DAY}")
  regionKey=$(hmac_sha256 hexkey:"${dateKey}" "${AWS_REGION}")
  serviceKey=$(hmac_sha256 hexkey:"${regionKey}" "${AWS_SERVICE}")
  signingKey=$(hmac_sha256 hexkey:"${serviceKey}" "aws4_request")

  printf "${stringToSign}" | openssl dgst -sha256 -mac HMAC -macopt \
      hexkey:"${signingKey}" | awk '{print $2}'
}

readonly SIGNATURE="$(create_signature)"

readonly HTTP_REQUEST_AUTHORIZATION_HEADER="\
AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY_ID}/${CURRENT_DATE_DAY}/\
${AWS_REGION}/${AWS_SERVICE}/aws4_request, \
SignedHeaders=${HTTP_REQUEST_SIGNED_HEADERS};x-amz-date, Signature=${SIGNATURE}"

curl -X "${HTTP_REQUEST_METHOD}" -v \
    "https://${AWS_SERVICE_ENDPOINT_URL}${HTTP_CANONICAL_REQUEST_URI}" \
    -H "Authorization: ${HTTP_REQUEST_AUTHORIZATION_HEADER}" \
    -H "content-type: ${HTTP_REQUEST_CONTENT_TYPE}" \
    -H "x-amz-content-sha256: ${HTTP_REQUEST_PAYLOAD_HASH}" \
    -H "x-amz-date: ${CURRENT_DATE_ISO8601}"

但請注意，如果您沒有理由創建簽名，最好使用 AWS API。

Answer 6

一個真正有效的解決方案（2021 年 12 月生效）是使用此腳本。 只需要在使用前導出密鑰（或將其值復制到 .sh 文件）。

export AWS_ACCESS_KEY_ID=AKxxx
export AWS_SECRET_ACCESS_KEY=zzzz

要下載只需運行

./s3download.sh get s3://mybucket/myfile.txt myfile.txt

您只需要傳遞get 、s3 存儲桶以及文件名和輸出文件。

需要腳本

創建一個s3download.sh文件和chmod +x s3download.sh ，用於上面的命令。

#!/bin/bash
set -eu
s3simple() {
  local command="$1"
  local url="$2"
  local file="${3:--}"

  # todo: nice error message if unsupported command?

  if [ "${url:0:5}" != "s3://" ]; then
    echo "Need an s3 url"
    return 1
  fi
  local path="${url:4}"

  if [ -z "${AWS_ACCESS_KEY_ID-}"  ]; then
    echo "Need AWS_ACCESS_KEY_ID to be set"
    return 1
  fi

  if [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
    echo "Need AWS_SECRET_ACCESS_KEY to be set"
    return 1
  fi

  local method md5 args
  case "$command" in
  get)
    method="GET"
    md5=""
    args="-o $file"
    ;;
  put)
    method="PUT"
    if [ ! -f "$file" ]; then
      echo "file not found"
      exit 1
    fi
    md5="$(openssl md5 -binary $file | openssl base64)"
    args="-T $file -H Content-MD5:$md5"
    ;;
  *)
    echo "Unsupported command"
    return 1
  esac

  local date="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"
  local string_to_sign
  printf -v string_to_sign "%s\n%s\n\n%s\n%s" "$method" "$md5" "$date" "$path"
  local signature=$(echo -n "$string_to_sign" | openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" | openssl base64)
  local authorization="AWS ${AWS_ACCESS_KEY_ID}:${signature}"

  curl $args -s -f -H Date:"${date}" -H Authorization:"${authorization}" https://s3.amazonaws.com"${path}"
}

s3simple "$@"

您可以在此處找到有關 s3simple 腳本的更多信息。