[英]How do I know if my Foundry job is using incremental computation?
I want to know if a job I'm debugging is using incremental computation or not since it's necessary for my debugging techniques.我想知道我正在调试的工作是否使用增量计算,因为这对于我的调试技术是必需的。
There are two ways to tell: the job's Spark Details will indicate this (if it's using Python), as will its code.有两种方法可以说明:作业的 Spark 详细信息将表明这一点(如果它使用 Python),它的代码也将表明这一点。
If you navigate to the Spark Details page as noted here , you'll notice there's a tab for Snapshot / Incremental
.如果您导航到此处所述的 Spark 详细信息页面,您会注意到有一个用于
Snapshot / Incremental
的选项卡。 In this tab, and if your job is using Python , you'll get a description of if your job is running using incremental computation.在此选项卡中,如果您的作业正在使用 Python ,您将获得有关您的作业是否正在使用增量计算运行的描述。 If the page reports
No Incremental Details Found
and you ran the job recently, this means it is not using Incremental computation .如果页面报告
No Incremental Details Found
并且您最近运行了该作业,这意味着它没有使用增量计算。 However, if your job is somewhat old (typically more than a couple of days), this may not be correct as these Spark details are removed for retention reasons.但是,如果您的工作有些旧(通常超过几天),这可能不正确,因为这些 Spark 详细信息会因保留原因而被删除。
A quick way to check if your job's information has been removed due to retention is to navigate to the Query Plan tab and see if any information is present.检查您的作业信息是否因保留而被删除的一种快速方法是导航到“查询计划”选项卡并查看是否存在任何信息。 If nothing is present, this means your Spark details have been deleted and you will need to re-run your job in order to see anything.
如果什么都不存在,这意味着您的 Spark 详细信息已被删除,您需要重新运行您的作业才能看到任何内容。 If you want a more reliable way of determining if a job is using incremental computation, I'd suggest following the second method below.
如果您想要一种更可靠的方法来确定作业是否使用增量计算,我建议您遵循下面的第二种方法。
If you navigate to the code backing the Transform, you'll want to look for a couple indicators, depending on the language used.如果您导航到支持转换的代码,您将需要查找几个指标,具体取决于所使用的语言。
The Transform will have an @incremental()
decorator on it if it's using incremental computation.如果使用增量计算,Transform 将有一个
@incremental()
装饰器。 This doesn't indicate however whether it will choose to write or read an incremental view.然而,这并不表示它是否会选择写入或读取增量视图。 The backing code can choose what types of reads or writes it wishes to do, so you'll want to inspect the code more closely to see what it's written to do.
支持代码可以选择它希望执行的读取或写入类型,因此您需要更仔细地检查代码以了解其写入的目的。
from transforms.api import transform, Input, Output, incremental
@incremental() # This being present indicates incremental computation
@transform(...)
def my_compute_function(...):
...
The Transform will have the getReadRange
and getWriteMode
methods overridden in the backing code. Transform 将在支持代码中覆盖
getReadRange
和getWriteMode
方法。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.