如何在 Glue 作业中访问 AWS Glue 工作流的运行属性？

Question

I have been working with AWS Glue workflow for orchestrating batch jobs. we need to pass push-down-predicate in order to limit the processing for batch job. When we run Glue jobs alone, we can pass push down predicates as a command line argument at run time (ie aws glue start-job-run --job-name foo.scala --arguments --arg1-text ${arg1}..). But when we use glue workflow to execute Glue jobs, it is bit unclear.

When we orchestrate Batch jobs using AWS Glue workflows, we can add run properties while creating workflow.

1. Can I use run properties to pass push down predicate for my Glue Job?
2. If yes, then how can I define value for the run property (push down predicate) at run time. The reason I want to define value for push down predicate at run time, is because the predicate arbitrarily changes every day. (ie run glue-workflow for past 10 days, past 20 days, past 2 days etc.)

I tried:

aws glue start-workflow-run --name workflow-name | jq -r '.RunId '

aws glue put-workflow-run-properties --name workflow-name --run-id "ID" --run-properties --pushdownpredicate="some value"

I am able to see the run property I have passed using put-workflow-run-property

aws glue put-workflow-run-properties --name workflow-name --run-id "ID"

But I am not able to detect "pushdownpredicate" in my Glue Job. Any idea how to access workflow's run property in Glue Job?

Answer 1

If you are using python as programming language for your Glue job then you can issue get_workflow_run_properties API call to retrieve the property and use it inside your Glue job.

response = client.get_workflow_run_properties(
    Name='string',
    RunId='string'
)

This will give you below response which you can parse and use it:

{
    'RunProperties': {
        'string': 'string'
    }
}

If you are using scala then you can use equivalent AWS SDK.

Answer 2

In first instance you need to be sure that the job is running from a workflow:

def get_worfklow_params(args: Dict[str, str]) -> Dict[str, str]:
    """
    get_worfklow_params is delegated to retrieve the WORKFLOW parameters
    """
    glue_client = boto3.client("glue")
    if "WORKFLOW_NAME" in args and "WORKFLOW_RUN_ID" in args:
        workflow_args = glue_client.get_workflow_run_properties(Name=args['WORKFLOW_NAME'], RunId=args['WORKFLOW_RUN_ID'])["RunProperties"]
        print("Found the following workflow args: \n{}".format(workflow_args))
        return workflow_args
    print("Unable to find run properties for this workflow!")
    return None

This method will return a map of the workflow input parameter.

Than you can use the following method in order to retrieve a given parameter:

def get_worfklow_param(args: Dict[str, str], arg) -> str:
    """
    get_worfklow_param is delegated to verify if the given parameter is present in the job and return it. In case of no presence None will be returned
    """
    if args is None:
        return None
    return args[arg] if arg in args else None

From reuse the code, in my opinion is better to create a python ( whl ) module and set the module in the script path of your job. By this way, you can retrieve the method with a simple import.

Without the whl module, you can move in the following way:

def MyTransform(glueContext, dfc) -> DynamicFrameCollection:
    import boto3
    import sys
    from typing import Dict

    def get_worfklow_params(args: Dict[str, str]) -> Dict[str, str]:
    """
    get_worfklow_params is delegated to retrieve the WORKFLOW parameters
    """
    glue_client = boto3.client("glue")
    if "WORKFLOW_NAME" in args and "WORKFLOW_RUN_ID" in args:
        workflow_args = glue_client.get_workflow_run_properties(
            Name=args['WORKFLOW_NAME'], RunId=args['WORKFLOW_RUN_ID'])["RunProperties"]
        print("Found the following workflow args: \n{}".format(workflow_args))
        return workflow_args
    print("Unable to find run properties for this workflow!")
    return None

    def get_worfklow_param(args: Dict[str, str], arg) -> str:
    """
    get_worfklow_param is delegated to verify if the given parameter is present in the job and return it. In case of no presence None will be returned
    """
    if args is None:
        return None
    return args[arg] if arg in args else None

    _args = getResolvedOptions(sys.argv, ['JOB_NAME', 'WORKFLOW_NAME', 'WORKFLOW_RUN_ID'])
    worfklow_params = get_worfklow_params(_args)


    job_run_id = get_worfklow_param(_args, "WORKFLOW_RUN_ID")
    my_parameter= get_worfklow_param(_args, "WORKFLOW_CUSTOM_PARAMETER")

Answer 3

If you run Glue Job using workflow then sys.argv (which is a list) will contain parameters --WORKFLOW_NAME and --WORKFLOW_RUN_ID in it. You can use this fact to check if a Glue Job is being run by Workflow or not and then retrieve the Workflow Runtime Properties

from awsglue.utils import getResolvedOptions

if '--WORKFLOW_NAME' in sys.argv and '--WORKFLOW_RUN_ID' in sys.argv:
    glue_args = getResolvedOptions(
        sys.argv, ['WORKFLOW_NAME', 'WORKFLOW_RUN_ID']
    )

    workflow_args = glue_client.get_workflow_run_properties(
        Name=glue_args['WORKFLOW_NAME'], RunId=glue_args['WORKFLOW_RUN_ID']
    )["RunProperties"]

    return {**workflow_args}
else:
    raise Exception("GlueJobNotStartedByWorkflow")