[英]How to resume a Prefect flow on failure without having to re-run the entire flow?
I wasn't able to use prefect's FlowRunner to solve the above question.我无法使用 prefect 的FlowRunner来解决上述问题。 I likely either used it wrong (see below) or missed something.我可能要么用错了(见下文),要么错过了一些东西。 Would really appreciate any pointers!真的很感激任何指点!
I read through the fantastic prefect core documentation and found the sections on Handling Failure and Local Debugging to be the most relevant to this (may have missed something.).我通读了出色的核心文档,发现 处理失败和本地调试的部分与此最相关(可能遗漏了一些东西。)。 The FlowRunner class appeared (to me) to be the solution. FlowRunner class 似乎(对我来说)是解决方案。
To see if I could use Flow Runner to resume a failed flow:看看我是否可以使用 Flow Runner 来恢复失败的流程:
from time import sleep
import prefect
from prefect import Flow, task
@task
def success():
sleep(3)
return
@task
def failure():
return 1 / 0
def get_flow_runner():
with Flow("Success/Failure") as flow:
success()
failure()
return prefect.engine.FlowRunner(flow)
In [1]: run nameofscript.py
In [2]: flow_runner = get_flow_runner()
In [3]: state = flow_runner.run()
Replaced 1 / 0 with 1 / 1 in failure()
so task would be successful:在failure()
中将 1 / 0 替换为 1 / 1 以便任务成功:
And finally passed the previous state to the flow_runner
hoping that it would resume the flow:最后将之前的 state 传递给flow_runner
,希望它能够恢复流程:
In [1]: run nameofscript.py
In [2]: flow_runner = get_flow_runner()
In [3]: flow_runner.run(task_states=state.result)
The entire flow ran again including the 3 second successful task.整个流程再次运行,包括 3 秒成功的任务。
The issue here is that you are rebuilding your Flow with each run, which changes the Task objects.这里的问题是您在每次运行时都在重建 Flow,这会更改 Task 对象。 state.result
is a dictionary whose keys are Task objects - if the underlying Task object changes in any way, so will its hash. state.result
是一个字典,其键是任务对象 - 如果基础任务 object 发生任何变化,那么它的 hash 也会发生变化。 You should instead manually create the dictionary of states using the updated Task objects, like so:您应该改为使用更新的 Task 对象手动创建状态字典,如下所示:
from prefect.engine.state import Success
failure_task = runner.flow.get_tasks(name="failure")[0]
task_states = {failure_task: Success("Mocked success")}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.