AWS Glue：由于缺少元数据而无法启动作业运行

Question

In order to run a job using boto3, the documentation states only JobName is required. However, my code:

    def start_job_run(self, name):
        print("The name of the job to be run via client is: {}".format(name))
        self.response_de_start_job = self.client.start_job_run(
            JobName=name
        )
        print(self.response_de_start_job)

and the client is:

    self.client = boto3.client(
            'glue',
            region_name='ap-south-1',
            aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
            aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY'),
        )

when executed via Python3, gives out an error:

botocore.errorfactory.EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the StartJobRun operation: Failed to start job run due to missing metadata

but when I do the same operation on the same job from UI and from the cli( aws glue start-job-run --job-name march15_9 ), it works all alright.

Answer 1

In my experience the error often means Can not find the job . As soon as jobs are bound to regions, a combination of name and region uniquely identifies a job, and errors in any of these (including trivial mistypes) will lead to the error you experience(d). As an example, a job I am using is in the us-east-1, thus the following statement executes successfully.

    glue_client = boto3.client('glue', region_name='us-east-1')
    response = glue_client.start_job_run(
        JobName = glue_job_name)

However, the snippet below will produce the same error as you have

    glue_client = boto3.client('glue', region_name='us-west-1')
    response = glue_client.start_job_run(
        JobName = glue_job_name)

botocore.errorfactory.EntityNotFoundException: An error occurred (EntityNotFoundException) when calling the StartJobRun operation: Failed to start job run due to missing metadata

In the case above, it's relatively easy to check, by specifying running cli with --region parameter

It would be something like: aws glue start-job-run --job-name march15_9 --region ap-south-1

If this runs successfully(thus the region is indeed ap-south-1), I'd explicitly set parameters in the code to remove unknown factors, and instead of passing them through environment variables you can temporary put string values in the code.

Once the code works with hardcoded values you may remove them one by one, thus finding one (or a few) that need to passed correctly.

All the best

PS Indeed, the documentation is correct, only JobName needs to be set as a parameter, I have code that works this way

Answer 2

What glue error log indicating?

You may be using some parameters in glue job which you are not passing while calling job

Answer 3

I too faced the same error, the problem is passing ARN of glue job as JobName. Resolved by passing only Name of the glue job.

response = client.start_job_run(
    JobName='Glue Job Name not ARN'
)

Answer 4

Check if the name of your glue job is correctly written. I had a similar case and I fixed it that way. (For example: Job_ 01 instead of Job_01)