如何检索azure数据工厂管道的json表示？

Question

I want to track pipeline changes in source control, and I'm looking for a way to programmatically retrieve the json representation from the ADF.

The .Net routines return the objects, but sadly ToString() does not return json (wouldn't THAT be convenient?), so right now I'm looking at copying the json down by hand (shoot me now!), or possibly trying to recreate the json from the .Net objects (shoot me later!).

Please tell me I'm being dense and there is an obvious way to do this.

Answer 1

You can serialize the object using Newtonsoft Json. See ( https://azure.microsoft.com/en-us/documentation/articles/data-factory-create-data-factories-programmatically/ ) for how to connect via the ADF SDK

var aadTokenCredentials = new TokenCloudCredentials(ConfigurationManager.AppSettings["SubscriptionId"], GetAuthorizationHeader());

var resourceManagerUri = new Uri(ConfigurationManager.AppSettings["ResourceManagerEndpoint"]);
var manager = new DataFactoryManagementClient(aadTokenCredentials, resourceManagerUri);
var pipeline = manager.Pipelines.Get(resourceGroupName, dataFactoryName, pipelineName);
var pipelineAsJson = JsonConvert.SerializeObject(pipeline.Pipeline, Formatting.Indented);

I was expecting something more complex but looking at the sdk source GitHub it is not doing anything special.

Answer 2

Our team has a deployment tool that takes git changes and deploy them appropriately. Everything is done asynchronously and being controlled and versioned through git. In a nutshell our deployment has the following flow:

• Any completed git merge request triggers a VSO build. This is simply building the whole solution via MsBuild.

• Every successful build is applied a Git tag for tracking of Last Known Good.

• Next (if build succeeded) our .net ADFPublisher starts by taking only the changed data factory files and asynchronously publishing them based on their git operation (modified, add, delete, etc.).

• For some failures cases our ADFPublisher will perform a retry.

  This whole process (Build + publish) takes ~ 65 seconds and has already saved us from having several bugs. It also allows us to move definitions from one environment to another very easily.

Let me know if you think this is something that you will be interested in and I will setup a way to share it with you