Azure Data Factory Pipeline - Store single-value source query output as a variable to then use in Copy Data activity

Question

I am looking to implement an incremental table loading pipeline in ADF. I want to execute a query to get the latest timestamp from the table in an Azure SQL database. Then, store this value as a variable in ADF so I can then reference it in the "Source" query of a Copy Data activity.

The goal is to only request data from an API with a timestamp greater than the latest timestamp in the SQL table.

Is this functionality possible within ADF pipelines? or do I need to look to Azure functions or Data Flows?

Answer 1

This is definitely possible with Data Factory. You could use the Lookup Activity or a Stored Procedure, but the team just released the new Script Activity:

This will return results like so:

{
    "resultSetCount": 1,
    "recordsAffected": 0,
    "resultSets": [
        {
            "rowCount": 1,
            "rows": [
                {
                    "MaxDate": "2018-03-20"
                }
            ]
...
}

Here is the expression to read this into a variable:

@activity('Script1').output.resultSets[0].rows[0].MaxDate