无法在 Lambda 中创建 AWS 胶水爬虫，Lambda 由步骤 function 触发

Question

This is my scenario:

1. API gateway/Lambda triggers a Step function. The payload to the step function is a SQL query to run on an existing Athena table

2. Task-1 of Step function calls StartQueryExecution API on Athena. The query runs successfully and generates results in a given S bucket

3. Task-2 invokes a Lambda which creates an AWS crawler based on the results from Task-2 (Task-2 gets the S3 file location as input from Task-1)

4. Task-3 invokes a Lambda that runs the crawler that is created in Task-2

To create AWS crawler in Lambda, following is my code in Lambda (NodeJS):

  exports.handler = async(event) => {
    const awsglue = new aws.Glue();

    const uuid = event.QueryExecutionID
    var path = event.OutputPath

    var params = {
        Name: uuid,
        Role: <Role ARN>,
        DatabaseName: <Database name>,
        Targets: {
            S3Targets: [{
                Path: path
            }]
        }
    }

   var request = await awsglue.createCrawler(params, (err, data) => {
   if (err) console.log(err, err.stack);
   else console.log(data);
    })
  
    const response = {
        statusCode: 200,
        body: JSON.stringify(uuid),
    };
    return response;
};

Problem createCrawler being an asynchronous call, Lambda returns SUCCESS, even before the creation of the crawler. Consequently, Task-3, which is supposed to run the crawler, fails.

To work around this problem I tried combining createCrawler and startCrawler in the same Lambda function but that too doesn't work.

Am I missing something? Is it not possible to create an AWS Glue crawler in a Lambda function that is triggered by a Step function?

Answer 1

The Glue crawler should be created with the state machine of Step Functions managed by Infra as code, like CloudFormation, Terraform and AWS CDK.

Then the lambda func start the crawler and retrieve the result of crawler's run.

See the similar sample code,

Lambda Func to start crawler, https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/blob/61b341eb1fed1b1b471c9fdccce0c348ff7f343f/src/lambda.d/crawl-data-catalog/index.ts#L24-L37

Create crawler by CDK, https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/blob/61b341eb1fed1b1b471c9fdccce0c348ff7f343f/src/lib/etl-glue.ts#L78-L95