Pulling job details for an Azure Data Factory Job from SQL Server

Question

I have a SQL Server database in the cloud that will have a table with unprocessed jobs with their parameters. I'd like to have an Azure Data Factory pipeline that checks that table every fifteen minutes, gets the list of jobs, executes them in a loop, and then marks them as complete in the table.

I'm not seeing anything like this loop structure in ADF - am I overlooking something?

Answer 1

We have added rich control flow constructs in ADF V2 (currently in Public Preview) to enable the scenario you described above. Specifically you can use a scheduled trigger executing a lookup activity followed by a ForEach activity, execute the job, and on success execute a Stored Procedure activity to mark it as success.

There is a tutorial that illustrates how to use LookUp+ForEach+Copy that you can use and modify to fit your specific scenario: https://docs.microsoft.com/en-us/azure/data-factory/tutorial-bulk-copy

Answer 2

Maybe starting with this quickstart guide is more approachable? https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-powershell This particular guide uses Powershell, but if you prefer REST API, .NET SDK, or Python SDK, you can find those in the left navigation of the documentation table of contents.

We are also actively working on a GUI-based authoring app which will enable easy drag-n-drop experience to build out pipelines contained multiple chained activities. We will post announcements in blogs once this feature becomes available.