从 Airflow 触发 AWS Lambda function

Question

I have created a function in AWS lambda which looks like this:

import boto3
import numpy as np
import pandas as pd
import s3fs
from io import StringIO


def test(event=None, context=None):
    # creating a pandas dataframe from an api
    # placing 2 csv files in S3 bucket

This function queries an external API and places 2 csv files in S3 bucket. I want to trigger this function in Airflow, I have found this code:

import boto3, json, typing

def invokeLambdaFunction(*, functionName:str=None, payload:typing.Mapping[str, str]=None):
    if  functionName == None:
        raise Exception('ERROR: functionName parameter cannot be NULL')
    payloadStr = json.dumps(payload)
    payloadBytesArr = bytes(payloadStr, encoding='utf8')
    client = boto3.client('lambda')
    response = client.invoke(
        FunctionName=test,
        InvocationType="RequestResponse",
        Payload=payloadBytesArr
    )
    return response



if __name__ == '__main__':
    payloadObj = {"something" : "1111111-222222-333333-bba8-1111111"}
    response = invokeLambdaFunction(functionName='test',  payload=payloadObj)
    print(f'response:{response}')

But as I understand this code snippet does not connect to the S3. Is this the right approach to trigger AWS Lambda function from Airflow or there is a better way?

Answer 1

I would advice to use the AwsLambdaHook :

https://airflow.apache.org/docs/stable/_api/airflow/contrib/hooks/aws_lambda_hook/index.html#module-airflow.contrib.hooks.aws_lambda_hook

And you can check a test showing its usage to trigger a lambda function:

https://github.com/apache/airflow/blob/master/tests/providers/amazon/aws/hooks/test_lambda_function.py