How to get output parameter from Executed Pipeline in ADF?

Question

I have a databricks pipeline that will give an output, but at the moment, I need run the databricks from the Executed Pipelines, when I tried to run it, my databricks output didn't show up on Executed Pipelines ? Is this pipeline can't show the output ?

So this is my Databricks output result.

And this is my Executed Pipeline.

How can I get the runOutput result from Executed Pipeline ?

Answer 1

Data bricks activity is inside child pipeline. So execute pipeline activity will not show output properties of child pipeline activites.

You can try this work around, You can write Child pipeline's output to a database table. Output means A blob name or writing the parent runID to a SQL table.

By doing this parent pipeline will get reference to get the output.

Answer 2

You can use the Azure REST API to query activity runs, so you could do an API call to get the last pipeline run of the pipeline you are interested in. Then you use those details to call another REST API to get the Activity output for that pipeline RunID you are interested in.

This method has the benefit of not needing to write output to a file or DB, rather you can just lookup the output of that activity in the Azure Monitor Logs for ADF. It can also be used for getting activity output from a completely different Data Factory.

You can actually refer to this post to show how to use the Azure REST API for ADF to query pipeline and activity runs:

Azure data factory and Log analytics

The important part here is the body and the operands:

Here is the Microsoft Documentation on how to use the query pipeline API:

https://docs.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory

And here is the Microsoft documentation on how to query the activity API:

https://docs.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory

So what you can do is find the Pipeline RunID of the child pipeline that ran using the pipeline API, then use that to query for the specific activity execution within that pipeline using the Activity API.

Real Example:

Parent Pipeline:

Child Pipeline:

To get the Last Pipeline Run of the Child Pipeline using Web Request:

URL:

https://management.azure.com/subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01

BODY:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [
    {
      "operand": "PipelineName",
      "operator": "Equals",
      "values": [
        "@{pipeline().parameters.PipelineName}"
      ]
    },
    {
      "operand": "LatestOnly",
      "operator": "Equals",
      "values": [
        true
      ]
    }
  ]
}

To get the Activity Output using the Pipeline details from Previous Web call:

URL:

https://management.azure.com/subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/pipelineruns/@{activity('Get Last ChildPipeline Run Details').output.value[0].runId}/queryActivityruns?api-version=2018-06-01

BODY:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [
    {
      "operand": "ActivityName",
      "operator": "Equals",
      "values": [
        "@{pipeline().parameters.ActivityName}"
      ]
    }
  ]
}

Here is the successful output in the Parent Pipeline using this process shown above: