如何在作业 AWS Glue 中获取开始和结束时间？

Question

I want to get Start and End time in a Job AWS Glue, i tested these options but it doesn't work.

args = getResolvedOptions(sys.argv, ['TempDir','JOB_NAME','START_JOB_RUN','END_JOB_RUN'])
start_job_run = args['START_JOB_RUN']

or

args = getResolvedOptions(sys.argv, ['TempDir','JOB_NAME'])
job_name = args['JOB_NAME']
start_job_run = job_name['START_JOB_RUN']

Another option, but in my opinion it's not the best way, it's define at the beginning of script start time and in the end a end time.

Answer 1

You can't get the job start and end time within the same glue job.

One option is that you can always use the system time to log when the job started and when it ended.

Otherwise, you should use the JobRun API to get the job metadata and status.

You can use boto3 library to retrive the details.

response = glueClient.get_job_run(
    JobName='string',
    RunId='string',
    PredecessorsIncluded=True|False
)

It will result in startTime, endTime along with other details.

Here is the detailed documentation:

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_run
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-runs.html