How to trigger an alert notification of a long-running process in Azure Data Factory V2 using either Azure Monitor or ADF itself?

Question

I've been trying to find the best way to trigger an alert when an ADF task (ie CopyActivity or Stored Procedure Task) has been running for more than N hours, I wanted to use the Azure Monitor as it is one of the recommended notification services in Azure, however I have not been able to find a "Running" criteria, hence I had to play with the available criteria (Succeeded and Failed) and check this every N hours, however this is still not perfect as I don't know when the process started and we may run the process manually multiple times a day, is there any way you would recommend doing this? like a event-based notification that listens to some time variable and as soon as it is greater than the threshold triggers an email notification?

Answer 1

is there any way you would recommend doing this? like a event-based notification that listens to some time variable and as soon as it is greater than the threshold triggers an email notification?

Based on your requirements, I suggest you using Azure Data Factory SDKs to monitor your pipelines and activities.

You could create a time trigger Azure Function which is triggered every N hours. In that trigger function :

1. You could list all running activities in data factory account.

2. Then loop them to monitor the DurationInMs Property in ActivityRun Class to check if any activity has been running for more than N hours and it's still In-Progress status .

3. Finally, send the email or kill the activity or do whatever you want.

Answer 2

I would suggest simple solution: Kusto query for listing all pipeline runs where status is "Queued" and joining it on CorrelationId with those that we are not interested in - typically "Succeeded", "Failed". Join flavor leftanti does the job by "Returning all the records from the left side that don't have matches from the right." (as specified in MS documentation). Next step would be to set your desired timeout value - it is 30m in the example code below. Finally, you can configure Alert rule based on this query and get your email notification, or whatever you need.

ADFPipelineRun | where Status == "Queued" | join kind=leftanti ( ADFPipelineRun | where Status in ("Failed", "Succeeded") ) on CorrelationId | where Start < ago(30m)

I tested this only briefly, maybe there is something missing. I have an idea about adding other statuses to be removed from result - like "Cancelled".