在 AWS Glue 交互中提供参数 session

Question

I am trying to develop an AWS Glue job in pyspark using interactive sessions in AWS Glue. To use job.init , I have seen in examples that the JOB_NAME is passed like so

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
...
job.init(args['JOB_NAME'], args)

a) When I run the notebook in the Glue Studio Editor, I get: GlueArgumentError: the following arguments are required: --JOB_NAME . How do I set the parameter without running the whole job?

b) Ultimately, I want to develop in VS Code. Is there a way to set the JOB_NAME parameter there?

Answer 1

In your Glue Interactive sessions notebook you can add required arguments manually by updating the sys.argv. See the example below:

sys.argv+=['--JOB_NAME', 'my_test_job']
args = getResolvedOptions(sys.argv,
                          ['JOB_NAME'])
print(args)