ADF Pipeline Missed Schedule by 15 minutes

Question

I am new to Azure Data Factory. With help from my colleagues we created a pipeline to import data from an external system. We scheduled a trigger to run every two hours (5AM, 7AM, 9AM... 11PM). The trigger ran 5AM and 7AM, but missed the 9AM run by 15 minutes. Could you please let me know if there is a particular reason for that? I have shared the screenshot.

Note: All times are in CST

Answer 1

The possible reason for this the pipeline could be in Queue waiting for resource to be available to execute the pipeline. This happens when multiple pipelines have been triggered at the same time and the job concurrent limit is full.

To overcome this issue, you need to increase the concurrent job limit.

The default value of the concurrent jobs limit is set based on the machine size. The factors used to calculate this value depend on the amount of RAM and the number of CPU cores of the machine. So the more cores and the more memory, the higher the default limit of concurrent jobs.

You scale out by increasing the number of nodes. When you increase the number of nodes, the concurrent jobs limit is the sum of the concurrent job limit values of all the available nodes. For example, if one node lets you run a maximum of twelve concurrent jobs, then adding three more similar nodes lets you run a maximum of 48 concurrent jobs (that is, 4 x 12). We recommend that you increase the concurrent jobs limit only when you see low resource usage with the default values on each node.

You can override the calculated default value in the Azure portal. Select Author > Connections > Integration Runtimes > Edit > Nodes > Modify concurrent job value per node .