How to invoke an AWS Lambda function from EC2 instances with python asyncio

Question

I recently posted a question about How to allow invoking an AWS Lambda function only from EC2 instances inside a VPC . I managed to get it working by attaching an IAM role with an "AWS lambda role" policy to the EC2 instances and now I can invoke the lambda function using boto3.

Now, I would like to make the call to the lambda function asynchronously using the asyncio await syntax. I read that the lambda function offers an asynchronous version by setting InvokeType='Event' , but that actually makes the call return immediately without getting the result of the function.

Since the function takes some time and I would like to launch many in parallel I would like to avoid blocking the execution while waiting for the function to return.

I tried using aiobotocore but that is only supporting basic 's3' service functionalities.

The best way to solve this (in humble opinion) would be to use the AWS API Gateway service to invoke the lambda function through a GET/POST request that can be easily handled using aiohttp.

Nevertheless I don't manage to make it work.

I added to the EC2 IAM role the policy "AmazonAPIGatewayInvokeFullAccess" but every time I try to:

import requests
r = requests.get('https://url_to_api_gateway_for_function')

I get a forbidden response <Response [403]> .

I created the API Gateway using directly the trigger in the lambda function.

I also tried to edit the API Gateway settings, by adding a post method to the function path and setting the "AWS_IAM" authentication and then deploying it as "prod" deployment...no luck. Still same forbidden response. When I test it through the "test screen on the API gateway, it works fine".

Any idea how to fix this? Am I missing some step?

Answer 1

I managed to solve my issue after some struggling.

The problem is that curl and python modules like python's requests do not sign the http requests with the IAM credentials of the EC2 machine where they are running. The http request to the AWS GATEWAY API must be signed using the AWS v4 signin protocol.

An example is here: http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

Luckily, to keep things simple, there are some helper modules like requests-aws-sign: https://github.com/jmenga/requests-aws-sign

At the end the code could look something like:

import aiohttp
import asyncio
from requests_aws_sign import AWSV4Sign
from boto3 import session

session = session.Session()
credentials = session.get_credentials()
region = session.region_name or 'ap-southeast-2'
service = 'execute-api'
url = "get_it_from_api->stages->your_deployment->invoke_url"
auth=AWSV4Sign(credentials, region, service)

async def invoke_func(loop):
    async with aiohttp.request('GET', url, auth=auth, loop=loop) as resp:
        html = await resp.text()
        print(html)

loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

Hope this will save time to somebody else!

EDIT:

For the sake of completeness and helping others I have to say that the code above does not work due to the fact that requests_aws_sign is not compatible with aiohttp. I was getting some "auth field error".

I manged to solve it by using:

async with session.get(url, headers=update_headers()) as resp:

where update_headers() is a simple function that mimics what requests_aws_sign was doing to the headers (so that I can then set them directly to the request above using the header parameter). It looks like this:

def update_headers(sim_id):
    url = urlparse("get_it_from_api->stages->your_deployment->invoke_url")
    path = url.path or '/'
    querystring = ''
    if url.query:
        querystring = '?' + urlencode(parse_qs(url.query), doseq=True)
    safe_url = url.scheme + '://' + url.netloc.split(':')[0] + path + querystring
    request = AWSRequest(method='GET', url=safe_url)
    SigV4Auth(credentials, service, region).add_auth(request)
    return dict(request.headers.items())