Azure 数据工厂 V2 多个环境，如 SSIS

Question

I'm coming from a long SSIS background, we're looking to use Azure data factory v2 but I'm struggling to find any (clear) way of working with multiple environments. In SSIS we would have project parameters tied to the Visual Studio project configuration (eg development/test/production etc...) and say there were 2 parameters for SourceServerName and DestinationServerName, these would point to different servers if we were in development or test.

From my initial playing around I can't see any way to do this in data factory. I've searched google of course, but any information I've found seems to be around CI/CD then talks about Git 'branches' and is difficult to follow.

I'm basically looking for a very simple explanation and example of how this would be achieved in Azure data factory v2 (if it is even possible).

Answer 1

It works differently. You create an instance of data factory per environment and your environments are effectively embedded in each instance.

So here's one simple approach:

1. Create three data factories: dev, test, prod
2. Create your linked services in the dev environment pointing at dev sources and targets
3. Create the same named linked services in test, but of course these point at your tst systems
4. Now when you "migrate" your pipelines from dev to test, they use the same logical name (just like a connection manager)

So you don't designate an environment at execution time or map variables or anything... everything in test just runs against test because that's the way the linked servers have been defined.

That's the first step.

The next step is to connect only the dev ADF instance to Git. If you're a newcomer to Git it can be daunting but it's just a version control system. You save your code to it and it remembers every change you made.

Once your pipeline code is in git, the theory is that you migrate code out of git into higher environments in an automated fashion.

If you go through the links provided in the other answer, you'll see how you set it up.

I do have an issue with this approach though - you have to look up all of your environment values in keystore, which to me is silly because why do we need to designate the test servers hostname everytime we deploy to test?

One last thing is that if you a pipeline that doesn't use a linked service (say a REST pipeline), I haven't found a way to make that environment aware. I ended up building logic around the current data factories name to dynamically change endpoints.

This is a bit of a bran dump but feel free to ask questions.

Answer 2

Although it's not recommended - yes, you can do it.
Take a look at Linked Service - in this case, I have a connection to Azure SQL Database:
You have possibilities to use dynamic content for either the server name and database name. Just add a parameter to your pipeline, pass it to the Linked Service and use in the required field.
Let me know whether I explained it clearly enough?

Answer 3

Yes, it's possible although not so simple as it was in VS for SSIS.
1) First of all: there is no desktop application for developing ADF, only the browser.
Therefore developers should make the changes in their DEV environment and from many reasons, the best way to do it is a way of working with GIT repository connected.
2) Then, you need "only":
a) publish the changes (it creates/updates adf_publish branch in git)
b) With Azure DevOps deploy the code from adf_publish replacing required parameters for target environment. I know that at the beginning it sounds horrible, but the sooner you set up an environment like this the more time you save while developing pipelines.

How to do these things step by step?
I describe all the steps in the following posts:
- Setting up Code Repository for Azure Data Factory v2
- Deployment of Azure Data Factory with Azure DevOps

I hope this helps.