Query a specific time range data from cosmos db and store it in sql database via azure data factory

Question

I am using Azure Data Factory V2 and want to copy the json data stored as documents from Azure cosmos db to a azure sql table, using a SQL query.

Situation: I want to query just the last hour json files from Cosmos db and store it in sql table. To do that, I create a new Trigger in my Data factory which recurrence the sql query each 1 hour.

Problem: At first I wrote a query that read all data from cosmos db and it work fine, but I want to query just the data which stored within the last hour in cosmos db.

What i need? In Sql, how to write a query to get just the new files which stored within the last hour in the cosmos db ?

Answer 1

You could define an udf in your cosmos db collection.

function getLastHourTime(){
    var date = new Date();
    var a = date.setHours(-1);
    return a;
}

Modify your sql to :

SELECT * FROM c where c.time >= udf.getLastHourTime()

Note: udf.getLastHourTime() returns Unix time stamp , you need to match the format.

Hope it helps you.