通过 Lambda 函数触发 AWS Glue 工作流

Question

I am new to AWS GLUE and trying to trigger Glue workflow using the Lambda function.

I am using the attribute boto3.client('glue') but I am getting an error saying :

Glue' object has no attribute start_workflow_run

Here is the piece of code that I am trying to run:

import json
import boto3
def lambda_handler(event, context):
client = boto3.client('glue')
client.start_workflow_run(Name = 'Workflow_New', Arguments = {})

Is there any other method by what I can achieve what I am trying to do?

Answer 1

Please refer to this SO on how to call AWS Glue from a lambda, with code snippet.

How to Trigger Glue ETL Pyspark job through S3 Events or AWS Lambda?

import boto3
print('Loading function')

def lambda_handler(event, context):
    source_bucket = event['Records'][0]['s3']['bucket']['name']
    s3 = boto3.client('s3')
    glue = boto3.client('glue')
    gluejobname = "YOUR GLUE JOB NAME"

    try:
        runId = glue.start_job_run(JobName=gluejobname)
        status = glue.get_job_run(JobName=gluejobname, RunId=runId['JobRunId'])
        print("Job Status : ", status['JobRun']['JobRunState'])
    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist '
              'and your bucket is in the same region as this '
              'function.'.format(source_bucket, source_bucket))
    raise e

Thanks

Yuva

Answer 2

this works to invoke a glue workflow from Lambda (python):

import json
import boto3
def lambda_handler(event, context):

    # add your region_name
    glue = boto3.client(service_name='glue', region_name='eu-west-2') 
    
    # only 'Name' parameter
    workflow_run_id = glue.start_workflow_run(Name = 'Your_Workflow')

    print(f'workflow_run_id: {workflow_run_id}')

https://docs.aws.amazon.com/glue/latest/dg/glue-dg.pdf

Answer 3

Please try using the below code snippet:

import boto3

glueClient = boto3.client('glue')

response = glueClient.start_workflow_run(Name = 'wf_name')

You can use also this documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.start_workflow_run

Answer 4

Following link you can find aws example and pass parameter for glue.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-calling.html

AWS Glue -- Access Workflow Parameters from Within Job

Answer 5

I don't think so that glue has function named 'start_workflow_run'. Please try 'start_job_run'

response = client.start_job_run(JobName = 'Workflow_New', Arguments = {} )

Answer 6

Try using:

import json
import boto3

def lambda_handler(event, context):
    glueClient = boto3.client('glue', region_name='us-west-2')
    response = glueClient.start_workflow_run(Name=Workflow_name)

Also, I think you might want to add error handling around the response as well!