如何使用 Boto3 库在 AWS Glue 中创建工作流程？

Question

I'm having problems to create a ETL workflow in AWS Glue using Boto3 library with all orchestration include. I created all jobs and triggers but when I use glue.create_workflow(Name="Test) command there's no way to include orchestration.

I tried to use this parameter

DefaultRunProperties={
        'string': 'string'
    }

With:

'--extra-files': 'file' #Like when you use a external libray in a job

'Graph': {'Nodes': [], 'Edges': []}} #Graph output using glue.get_workflow('Test')

Is this possible or you can only do this using AWS Console?

Answer 1

I had the same issue and did not find any solution directly from the Boto3 module.

However, I discovered AWS Glue Blueprint which allows you to create workflows from a template and therefore to duplicate them easily with a few input parameters. It is then possible to start this Blueprint from Boto3 by giving it the right inputs.

The official AWS documentation on this subject and their tutorial directory: https://docs.aws.amazon.com/glue/latest/dg/orchestrate-using-workflows.html https://github.com/awslabs/aws-glue-blueprint-libs

Hopefully this will help you