使用AWS Glue Job触发器来启动具有不同参数的作业

Question

I am using AWS Glue ETL scripts and triggers to run a number of jobs on data in s3. I have written a total of four jobs that will take specific parameters based on the data we want to run the jobs against. We want to be able to share the script for each of the jobs and pass in different parameters for the specific job that we want to run (ie job-A will have two different sets of parameters for different jobs-- data1 and data2. We set up a trigger to start job-B with data1 after job-A with data1 succeeds and a separate trigger to start job-B with data2 after job-A with data2 succeeds).

Looking into job triggers, however, we can create triggers that start a job on the previous job's success (ie when job-A with parameters data1 passed in succeeds, trigger job-B with parameters data1; when job-A with parameters data2 passed in succeeds, trigger job-B with parameters data2), but because we are sharing code, regardless of the parameters (data1 v. data2) passed in, if the jobA with parameters for data1 succeeds, because job-B is configured to run on success of job-A, two instances of job-B will be kicked off-- one with parameters data1 and one with parameters data2.

Ideally, we would like the triggers to only start the job with the matching set of parameters so we can share the glue ETL job scripts and only pass in parameters to triggers.

Is there a way we can achieve such a thing without creating different versions of the scripts?

Answer 1

I'm having a similar issue, and I'm afraid that at the moment the only solution is to manually trigger the job at the end of the other job (eg using boto3)

import boto3 

[...]

client = boto3.client('glue', 'us-east-1')
if data1:
    client.start_job_run(
            JobName='job-A',
            Arguments={
                '--data': data1)
elif data2:
    client.start_job_run(
            JobName='job-B',
            Arguments={
                '--data': data2)