ADF 管道在复制活动中添加顺序值

Question

Apologies if this has been asked and answered elsewhere. If it is, please do refer to the url in comments on replies. So here is the situation,

I am making an API Request, in response I get auth_token which I use in the Copy Activity as Authorization to retrieve data in JSON format and Sink it to Azure SQL Database. I am able to Map all the elements I'm receiving in JSON to the columns of Azure SQL Database. However, there are two columns ( UploadId and RowId ) that still need to be populated.

• UploadId is a GUID which will be same for the whole batch of rows (this I've managed to solve)
• RowId will be a sequence starting from 1 to end of that batch entry, and then for next batch (with new GUID value) it resets back to 1.

The database will look something like this,

| APILoadTime |      UploadId     |    RowId    |
|  2020-02-01 | 29AD7-12345-22EwQ |      1      |
|  2020-02-01 | 29AD7-12345-22EwQ |      2      |
|  2020-02-01 | 29AD7-12345-22EwQ |      3      |
|  2020-02-01 | 29AD7-12345-22EwQ |      4      |
|  2020-02-01 | 29AD7-12345-22EwQ |      5      |
--------------------------------------------------> End of Batch One / Start of Batch Two
|  2020-02-01 | 30AD7-12345-22MLK |      1      |
|  2020-02-01 | 30AD7-12345-22MLK |      2      |
|  2020-02-01 | 30AD7-12345-22MLK |      3      |
|  2020-02-01 | 30AD7-12345-22MLK |      4      |
|  2020-02-01 | 30AD7-12345-22MLK |      5      |
--------------------------------------------------> End of Batch Two and so on ... 

Is there a way in Azure Pipeline's Copy Activity to achieve this RowId behavior ... Or even if it's possible within Azure SQL Database.

Apologies for a long description, and Thank you in advance for any help! Regards

Answer 1

You need to use a Window Function to achieve this. ADF Data Flows have Window Transformation activities that are designed to do this exact thing.

Otherwise, you could load the data into a staging table and then use Azure SQL to window the data as you select it out like...

SELECT
    APILoadTime
    ,UploadId
    ,ROW_NUMBER() OVER (PARTITION BY UploadId ORDER BY APILoadTime) AS RowId
FROM dbo.MyTable;

Answer 2

Thanks a lot @Leon Yue and @JeffRamos, I've managed to figure out the solution, so posting it here for everyone else who might encounter the same situation,

The solution I found was to use a Stored Procedure within Azure Data Factory from where I call the Azure Data Flow Activity. This is the code I used for creating the RowId seed function,

CREATE PROCEDURE resetRowId
AS
BEGIN
    DBCC CHECKIDENT ('myDatabase', RESEED, 0)
END
GO

Once I have this Stored Procedure, all I did was something like this,

This does it for you, the reason I kept it 0 so that when new data comes in, it starts from 1 again ...

Hope this helps others too ...

Thank you all who helped in someway