是否可以创建将使用多个参数运行相同作业的 AWS Glue 工作流？

Question

Looks like I cannot add the same job to a workflow more than once but I need to run the same Glue job multiple times with different parameters. Not possible?

Answer 1

Yes you can always set properties on your workflow and then you can access them in your job as shown in below . Refer to this link for more information.

import sys
import boto3
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext

glue_client = boto3.client("glue")
args = getResolvedOptions(sys.argv, ['JOB_NAME','WORKFLOW_NAME', 'WORKFLOW_RUN_ID'])
workflow_name = args['WORKFLOW_NAME']
workflow_run_id = args['WORKFLOW_RUN_ID']
workflow_params = glue_client.get_workflow_run_properties(Name=workflow_name,
                                        RunId=workflow_run_id)["RunProperties"]

target_database = workflow_params['target_database']
target_s3_location = workflow_params['target_s3_location']

Now if you want properties to be changed for every run then you can do the same by using put_workflow_run_properties API call. This can be scheduled to run before your actual job starts it's execution so that the parameters are updated. This API call can be scheduled via lambda or as a part of your workflow which runs before your actual job starts it's execution.

Answer 2

Answer is... sort of. You can indeed start the same job with different parameters, however in a workflow, assuming you are waiting for the jobs to finish, you cannot identify the job by anything other than its name. So if you have job j that you run with params --a and --b , then run crawlers afterward, the predicate for the crawler only allows you to specify that you're waiting for job j , so cannot distinguish between the two instances.

However, keep in mind that the job is just a container for properties including the the underlying glue script(s) so you can create as many jobs as needed to call the same script with different parameters. Because they will have different names, this strategy is fine.

But still -- annoying!