[英]boto3 glue get_job_runs - check execution with certain date exists in the response object
I am trying to fetch glue job executions that got failed previous day using 'get_job_runs' function available through boto3's glue client.我正在尝试使用通过 boto3 的粘合客户端获得的“get_job_runs”function 获取前一天失败的粘合作业执行。 https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_runs . https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_job_runs 。
The request syntax, does not have an option to filter executions or job runs by date/status -请求语法没有按日期/状态过滤执行或作业运行的选项 -
response = client.get_job_runs(
JobName='string',
NextToken='string',
MaxResults=123
)
The response I receive back looks something like below -我收到的回复如下所示 -
{
"JobRuns": [
{
"Id": "jr_89bfa55b544f7eec4f6ea574dfb0345345uhi4df65e59869e93c5d8f5efef989",
"Attempt": 0,
"JobName": "GlueJobName",
"StartedOn": datetime.datetime(2021, 1, 27, 4, 32, 47, 718000, tzinfo=tzlocal()),
"LastModifiedOn": datetime.datetime(2021, 1, 27, 4, 36, 14, 975000, tzinfo=tzlocal()),
"CompletedOn": datetime.datetime(2021, 1, 27, 4, 36, 14, 975000, tzinfo=tzlocal()),
"JobRunState": "FAILED",
"Arguments": {
"--additional-python-modules": "awswrangler",
"--conf": "spark.executor.memory=40g",
"--conf ": "spark.driver.memory=40g",
"--enable-spark-ui": "true",
"--extra-py-files": "s3://GlueJobName/lttb.py",
"--job-bookmark-option": "job-bookmark-disable",
"--spark-event-logs-path": "s3://GlueJobName/glue-script/spark-event-logs"
},
"ErrorMessage": "MemoryError: Unable to allocate xxxxx",
"PredecessorRuns": [],
"AllocatedCapacity": 8,
"ExecutionTime": 199,
"Timeout": 2880,
"MaxCapacity": 8.0,
"WorkerType": "G.2X",
"NumberOfWorkers": 4,
"LogGroupName": "/aws-glue/jobs",
"GlueVersion": "2.0"
}
],
"NextToken": "string"
}
So, what I am doing now is looping through the response object to check if the "CompletedOn" date matches with yesterday's date using prev_day calculated using datetime and timedelta and I am doing this in a while loop to fetch last 10000 executions, as the 'get_job_runs' single call is insufficient.所以,我现在正在做的是通过响应 object 循环检查“CompletedOn”日期是否与使用 datetime 和 timedelta 计算的 prev_day 的昨天日期匹配,我在 while 循环中执行此操作以获取最后 10000 次执行,作为 ' get_job_runs 的单次调用是不够的。
import boto3
from datetime import datetime, timedelta
logger = logging.getLogger()
logger.setLevel(logging.INFO)
glue_client = boto3.client("glue")
def filter_failed_exec_prev_day(executions, prev_day) -> list:
filtered_resp = []
for execution in executions['JobRuns']:
if execution['JobRunState'] == 'FAILED' and execution['CompletedOn'].date() == prev_day:
filtered_resp.append(execution)
return filtered_resp
def get_final_executions() -> list:
final_job_runs_list = []
MAX_EXEC_SEARCH_CNT = 10000
prev_day = (datetime.utcnow() - timedelta(days=1)).date()
buff_exec_cnt = 0
l_job = 'GlueJobName'
response = glue_client.get_job_runs(
JobName=l_job
)
resp_count = len(response['JobRuns'])
if resp_count > 0:
buff_exec_cnt += resp_count
filtered_resp = filter_failed_exec_prev_day(response, prev_day)
final_job_runs_list.extend(filtered_resp)
while buff_exec_cnt <= MAX_EXEC_SEARCH_CNT:
if 'NextToken' in response:
response = glue_client.get_job_runs(
JobName=l_job
)
buff_exec_cnt += len(response['JobRuns'])
filtered_resp = filter_failed_exec_prev_day(response, prev_day)
final_job_runs_list.extend(filtered_resp)
else:
logger.info(f"{job} executions list: {final_job_runs_list}")
break
return final_job_runs_list
Here, I am using a while loop to break the call after hitting 10K executions, this is triple the amount of executions we see each day on the job.在这里,我使用 while 循环在达到 10K 次执行后中断调用,这是我们每天在作业中看到的执行量的三倍。
Now, I am hoping to break the while loop after I encounter execution that belongs to prev_day - 1
, so is it possible to search the entire response dict for prev_day - 1
to make sure all prev day's executions are covered considering the datetime.datetime object we receive from boto3 for CompletedOn
attribute?现在,我希望在遇到属于prev_day - 1
的执行后打破 while 循环,因此是否可以在整个响应字典中搜索prev_day - 1
以确保考虑到 datetime.datetime object 涵盖所有前一天的执行我们从 boto3 收到CompletedOn
属性?
Appreciate reading through.感谢通读。
Thank you谢谢
I looked at your code.我看了你的代码。 And I think it might return always the same result as you're not iterating through the resultset correctly.而且我认为它可能总是返回相同的结果,因为您没有正确地遍历结果集。 here:这里:
while buff_exec_cnt <= MAX_EXEC_SEARCH_CNT:
if 'NextToken' in response:
response = glue_client.get_job_runs(
JobName=l_job
)
you need to pass the NextToken value to the get_job_runs method, like this:您需要将 NextToken 值传递给 get_job_runs 方法,如下所示:
response = glue_client.get_job_runs(
JobName=l_job, NextToken= response['NextToken']
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.