boto3 - Step Functions generate_presigned_url InvalidSignatureException

Question

I'm attempting to generate a pre-signed URL for describe_execution .

After trying different variations of request methods, headers, requests bodies, etc. any request I send to the presigned url will always fail. Canonical Request Hex in the String-to-Sign never matches.

boto3 : 1.4.7

botocore : 1.7.44

Code

Note that I've replaced the values in the URLs for executionArn , X-Amz-Credential .

import boto3
import requests

sfn = boto3.client("stepfunctions")

presigned_url = sfn.generate_presigned_url(
  "describe_execution", 
  Params={
    "executionArn": "arn:aws:states:us-west-2:123:execution:abc:def-ghi-jkl-mno"
  }
)

response = requests.post(presigned_url, data={})
print(response.text)

Logs

DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable data_path from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable ca_bundle from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable api_versions from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_timeout from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_num_attempts from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.credentials:Looking for credentials via: env
DEBUG:botocore.credentials:Looking for credentials via: assume-role
DEBUG:botocore.credentials:Looking for credentials via: shared-credentials-file
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
DEBUG:botocore.loaders:Loading JSON file: /Users/maxwellgbrown/.virtualenvs/boto3/lib/python3.5/site-packages/botocore/data/endpoints.json
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.loaders:Loading JSON file: /Users/maxwellgbrown/.virtualenvs/boto3/lib/python3.5/site-packages/botocore/data/stepfunctions/2016-11-23/service-2.json
DEBUG:botocore.hooks:Event creating-client-class.stepfunctions: calling handler <function add_generate_presigned_url at 0x10f863c80>
DEBUG:botocore.args:The s3 config key is not a dictionary type, ignoring its value of: None
DEBUG:botocore.endpoint:Setting states timeout as (60, 60)
DEBUG:botocore.loaders:Loading JSON file: /Users/maxwellgbrown/.virtualenvs/boto3/lib/python3.5/site-packages/botocore/data/_retry.json
DEBUG:botocore.client:Registering retry handlers for service: stepfunctions
DEBUG:botocore.hooks:Event choose-signer.states.DescribeExecution: calling handler <function set_operation_specific_signer at 0x10f8dd2f0>
DEBUG:botocore.auth:Calculating signature using v4 auth.
DEBUG:botocore.auth:CanonicalRequest:
POST
/
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZZZ%2F20171115%2Fus-west-2%2Fstates%2Faws4_request&X-Amz-Date=20171115T191510Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-target&executionArn=arn%3Aaws%3Astates%3Aus-west-2%3123%3Aexecution%3Aabc%def-ghi-jkl-mno
content-type:application/x-amz-json-1.0
host:states.us-west-2.amazonaws.com
x-amz-target:AWSStepFunctions.DescribeExecution

content-type;host;x-amz-target
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG:botocore.auth:StringToSign:
AWS4-HMAC-SHA256
20171115T191510Z
20171115/us-west-2/states/aws4_request
3afe55bcd073fc24a7dce06bacc3661c000230fd2df13e40fbc4d2d6eb30849b
DEBUG:botocore.auth:Signature:
4bcb1c51cfcbf4274a17af7aa3202f23f6d7ceb314c2e1e1dd9faac1eb110e91
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): states.us-west-2.amazonaws.com
DEBUG:urllib3.connectionpool:https://states.us-west-2.amazonaws.com:443 "POST /?executionArn=arn%3Aaws%3Astates%3Aus-west-2%123%3Aexecution%3Aabc%def-ghi-jkl-mno&X-Amz-Expires=3600&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-target&X-Amz-Credential=ZZZ%2Fus-west-2%2Fstates%2Faws4_request&X-Amz-Date=20171115T191510Z&X-Amz-Signature=4bcb1c51cfcbf4274a17af7aa3202f23f6d7ceb314c2e1e1dd9faac1eb110e91 HTTP/1.1" 403 1030

print(response)

<InvalidSignatureException>
  <Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been
'POST
/
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=ZZZ%2Fus-west-2%2Fstates%2Faws4_request&amp;X-Amz-Date=20171115T191510Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-target&amp;executionArn=arn%3Aaws%3Astates%3Aus-west-2%3123%3Aexecution%3Aabc%3efg-hij-klm-nmo
content-type:
host:states.us-west-2.amazonaws.com
x-amz-target:

content-type;host;x-amz-target
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20171115T191510Z
20171115/us-west-2/states/aws4_request
1f390ad977de8b8bd1dd0226424bb1389ba62b6bb185784bf499a45b80c7d476'
</Message>
</InvalidSignatureException>

I've had no issues generating presigned urls for retrieving s3 objects doing this:

s3 = client('s3')
s3.generate_presigned_url('get_object', Params={'Bucket': 'my-bucket', 'Key': 'my/file.txt'})

Answer 1

Based off the response to my boto3 issue it seems like stepfunctions doesn't support presigned urls.

So with what I tried, I was able to get passed the signing problem by providing the headers in the request (which is what you have to do as the headers are not included in the url but are signed for). However, I ran into a serialization issue.

The tricky part with the generate_presigned_url() is that it is a general method that is applied to all services, but it is really up to the service in whether they actually support using presigned url's and this decision differs from service to service. Given that I got a serialization error back, it looks like step functions does not support using presigned url's.

Instead, I think to support the general functionality of generating/signing a request to defer making the HTTP request for later, we will need to expose a generate_presigned_request(), but for now I think you will have to use the client.describe_execution() method. Let us know what your thoughts on this.