Automated Deployment of ADF Pipelines using Azure DevOps CI/CD Pipelines

Question

I have automated the Azure ADF Pipeline Deployment process using Azure DevOps CI/CD pipelines with the help of https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (ie) Deploying pipelines from DEV to PROD environment ADF. I am using ARM Templates of the ADF to deploy pipelines from one environment to another. Hence I will be having a separate ARM_Parameter.json corresponding to each environment(Dev/Prod). The Problem is each ADF pipeline may have few base parameres along with it, which is not parameterized and hence it will not be available in parameter.json. Can you guys help me to replace the Dev Values with the PROD Values in Base Parameter section under each ADF Pipelines in an automated way during this automated ADF pipeline deployment process using CI/CD Pipelines?

Answer 1

I see two options:

1. If it's only for this RUN_ENVIRONMENT parameter, you could change your parameter to variable and use the system variable @pipeline().DataFactory to determine what environment you're running in.
2. Otherwist, you can configure the Data Factory to generate ARM Parameters for your pipeline parameter default values, but you'll have to create a custom arm-template-parameters-definition.json file. Check the documentation here

Answer 2

You could use Custom parameter with ARM template . The custom parameter for Pipeline could look like this:

"Microsoft.DataFactory/factories/pipelines": {
    "properties": {
        "parameters": {
                "RUN_ENVIRONMENT": "=:-:string"
        }
    }
},

Answer 3

Replace the Dev Values with the PROD Values in Base Parameter section

Based on your screenshot, the RUN_ENVIRONMENT is the parameter of pipeline, which means while convert to ARM template, its format like:

 "resources": [
    {
      ....
      ....
      "properties": {
        "parameters": {
          "RUN_ENVIRONMENT": {
            "type": "string",
            "defaultValue": "pro"
          }
        },...      
      },...
    }
  ]

It can not be replaced by using Override template parameters in ARM deploy task. Because it will prompt The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.

---

To around this error, just install one extension and add the Replace token task into the pipeline which before ARM deploy task. And this task will replace the value of content during the build runtime:

For how to apply this task in our pipeline, you could have a refer to my answer1 and answer2

Answer 4

There is another approach to publish ADF, from master (collaboration) branch. You can define (replace) value for every single node (property) in JSON file (ADF object). It will resolve your problem as you can provide a separate CSV config file per each environment (stage).

Example of CSV config file ( config-stage-UAT.csv ):

type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT

Then just run such cmdlet in PowerShell:

Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"

Check this out: azure.datafactory.tools (PowerShell module)