参数化 Azure 数据工厂中的连接（ARM 模板）

Question

I am trying to setup an Azure Data Factory in a CI/CD setup. I followed Microsoft's best practices ( https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment ).

I have 4 environments (dev, test, UAT, PRD)

In short what I have done:

• Create an Azure data factory and link it to my Azure DevOps repo on DEV environment

• Create an Azure data factory on the other environments (test, UAT and PRD), but do NOT link it to DevOps. Instead, pipelines are released on these data factories using ARM templates and release pipelines in Azure DevOps.

• I have parameterized all necessary parts to be able to overwrite the settings in each of my environments.

At this moment, I am able to succesfully deploy to my other environments, however, the linkedservice to my database on azure is not working. I have parameterized everything, like Microsoft suggests, but when I export my linkedservice to an ARM template, it uses a connection string instead of my parameterized settings.

Below is a picture of my settings in the Azure Data Factory portal:

When I try to export this to ARM templates, I get the following:

{
            "name": "[concat(parameters('factoryName'), '/database')]",
            "type": "Microsoft.DataFactory/factories/linkedServices",
            "apiVersion": "2018-06-01",
            "properties": {
                "parameters": {
                    "sqlServerUrl": {
                        "type": "string"
                    },
                    "databaseName": {
                        "type": "string"
                    },
                    "sqlPwd": {
                        "type": "string"
                    },
                    "sqlAdminUsername": {
                        "type": "string"
                    }
                },
                "annotations": [],
                "type": "AzureSqlDatabase",
                "typeProperties": {
                    "connectionString": {
                        "type": "SecureString",
                        "value": "[parameters('database_connectionString')]"
                    }
                }
            },
            "dependsOn": []
        },

The problem with this ARM templates is that it does not use the parameters to create the connection string, but it uses the connection string parameter database_connectionString (the connection string is by default always parameterized by Azure, so I cannot remove this parameter).

When the release pipeline uses this template, the connectionstring is not filled in (only the parameters are filled in) and hence, the connection to the database fails. How should you setup the connection so that you can automatically (without human interaction) deploy to all environments by only changing the parameters?

I do not want to change the ARM templates coming from Azure Data Factory, because this requires human interaction

Answer 1

One of the other answers provided is a valid way using override parameters. This answer will provide a different answer as well as some more context on how to define the SQL connections and how to utilize and implement some of the updates made with Key Vault and Data Factory Integration.

If using an on prem connection to SQL the conenction string will look like:

"Server={serverName};Database={databaseName};User ID={domain}\{userName};Password={password};Integrated Security=True;" 

The quotes are required and can be passed in as an override parameter.

If using an Azure Database or even using Key Vault look to add Managed Identity which Data Factory Supports by including this in your ARM template

 "identity": {
        "type": "SystemAssigned"
    }

Once this is added then the Azure SQL Database will need to have the managed identity added . This can be done via a reusable SQL Script like:

    DECLARE @rolename AS NVARCHAR(100) = 'DataFactory_User'
    DECLARE @username AS NVARCHAR(100) -- This will be the DataFactory name
    SET @username = 'DataFacotryName'

if exists(select * from sys.database_principals where name = @username and Type = 'X' or Type='E')
    BEGIN
        DECLARE @dropUserSql varchar(1000)
        SET @dropUserSql='DROP USER [' + @username + ']'
        PRINT 'Executing ' + @dropUserSql
        EXEC (@dropUserSql)
        PRINT 'Completed ' + @dropUserSql
    END

DECLARE @createUserSql varchar(1000)
SET @createUserSql='CREATE USER [' + @username + '] FROM EXTERNAL PROVIDER'
PRINT 'Executing ' + @createUserSql
EXEC (@createUserSql)
PRINT 'Completed ' + @createUserSql

I recommend dropping and recreating this user. SQL recognizes the thumbprint of the Managed Identity and everytime the DataFactory is dropped and recreated a new thumbprint is created.

In terms of leveraging Key Vault there is a LinkedService type of Key Vault that relies on the Managed Identity described above to retrieve secrets.

The Key Vault if deployed via ARM will need to have the access policy updated to something similar to this:

"accessPolicies": [
          {
            "tenantID": "[subscription().tenantId]",
            "objectId": "[reference(resourceId('Microsoft.DataFactory/factories/', parameters('DataFactoryName')), '2018-02-01', 'Full').identity.principalId]",
            "permissions": {
              "secrets": [
                "get"
              ],
              "keys": [
                "get"
              ],
              "certificates": [
                "import"
              ]
            },
            "dependsOn": [
              "[resourceId('Microsoft.DataFactory/factories/', parameters('DataFactoryName'))]"
            ]
          }
        ]

This snippet assumes the Key Vault and Data Factory are in the same ARM template. If they are not the access policy can still be accomplished via ARM by obtaining the ObjectId of the Data Factory defined Managed Identity and passing it in as the ObjectId and removing the dependsOn statement.

Answer 2

Just in case you haven't found an answer or solved this:

I was in a very similar situation, I had several Azure SQL databases that were parameterized (database name), and needed to change the SQL server name in my ADF Azure Release Pipeline to swap between environments.

I found that by inspecting the code definition of the Linked Service ({} next to it), you can get a connection string that includes any parameters that you have defined:

I copied the value of "connectionString" and modified what I needed to (server address) and left the parameters in place, and added it to the "connectionString" override parameter in my Azure Release Pipeline, and it worked! (include the quotes!)

I hope that this helps you or anyone in future suffering the same frustration/confusion.

Answer 3

I do this by using the Azure resource group deployment task in the Release. One of the options is Override template parameters which will allow you to do exactly what you need. You can create a space separated list of parameters to override your ARM Template and pass in a Variable

In your case it would be

-database_connectionstring $(VariableHere)

I would store the connection string in Azure KeyVault and link it to a Secure Variable Group. You can also just hit the padlock on a standard variable to secure it.

Then bind your custom variable to each Stage in your Release

Task

Override