How do I know if my Foundry job is using incremental computation?

Question

I want to know if a job I'm debugging is using incremental computation or not since it's necessary for my debugging techniques.

Answer 1

There are two ways to tell: the job's Spark Details will indicate this (if it's using Python), as will its code.

Spark Details

If you navigate to the Spark Details page as noted here , you'll notice there's a tab for Snapshot / Incremental . In this tab, and if your job is using Python , you'll get a description of if your job is running using incremental computation. If the page reports No Incremental Details Found and you ran the job recently, this means it is not using Incremental computation . However, if your job is somewhat old (typically more than a couple of days), this may not be correct as these Spark details are removed for retention reasons.

A quick way to check if your job's information has been removed due to retention is to navigate to the Query Plan tab and see if any information is present. If nothing is present, this means your Spark details have been deleted and you will need to re-run your job in order to see anything. If you want a more reliable way of determining if a job is using incremental computation, I'd suggest following the second method below.

Code

If you navigate to the code backing the Transform, you'll want to look for a couple indicators, depending on the language used.

Python

The Transform will have an @incremental() decorator on it if it's using incremental computation. This doesn't indicate however whether it will choose to write or read an incremental view. The backing code can choose what types of reads or writes it wishes to do, so you'll want to inspect the code more closely to see what it's written to do.

from transforms.api import transform, Input, Output, incremental

@incremental() # This being present indicates incremental computation
@transform(...)
def my_compute_function(...):
  ...

Java

The Transform will have the getReadRange and getWriteMode methods overridden in the backing code.