pipeline fails for stored Procedure called in Copy Activity - Azure data factory V2

Question

We've a SQL server stored procedure which returns the incremental records. If there are no changes to the table, then nothing is returned. Stored procedure does what is expected.

We're invoking the above said stored procedure via Copy activity in Azure data factory. It works fine for all the cases except when nothing (empty) is returned.

We are looking for an option, where Nothing(Empty) is returned from stored procedure, pipeline should skip and proceed further and also mark the whole pipeline successful rather failed.

Thanks

Answer 1

Your stored procedure needs to end by a SELECT, so it returns something - including an empty set if there is no rows to return.

However, to skip the pipeline if there is no row, DraganB's last answer is pretty relevant, I had to do that a couple of time on my current project.

Answer 2

As @DraganB said in the comment, activities could run in the flow,so you could do stored procedure activity --> if activity --> copy activity .If the output of sp activity is empty,then don't run the copy activity and end the pipeline.

Another idea, maybe you could learn about azure function external table trigger . You could add a status column in your table such as needToBeCopied , every insert or update operation will change the column. Then filter the data which need to be copied while running the copy activity.

Answer 3

It got resolved. The real issue was Copy activity wasn't returning the correct error message. There was an issue with access control.

Grant VIEW CHANGE TRACKING permission on a table to a user:
[sql]
GRANT VIEW CHANGE TRACKING ON OBJECT::dbo.databaselog to username
[/sql]

Incremental loading approach is through enabling Change Tracking on SQL Server Database and on required tables . Azure data factory should have logged error as 'Insufficient permissions on so and so table'. Instead it failed the whole pipeline with error message as 'Stored procedure might be invalid or stored procedure doesn't return any output'.

Anyway, we assigned the right permissions and issue got resolved. Now, it creates an Empty file just with header record in it when there's no output returned from Stored Procedure likewise in - Data Factory Avoiding creation of empty files