[英]How can you get valid JSON for Azure Data Factory V2 Datasets/Pipelines using PowerShell?
I have some pipelines in Azure Data Factory V2 that I need to update. 我在Azure Data Factory V2中有一些需要更新的管道。 Specifically I need to update the schema of the datasets.
具体来说,我需要更新数据集的架构。 I initially created them using the user interface.
我最初使用用户界面创建它们。 Since the number of columns is very large, the user interface has become very slow for working on them, so I'd like to switch to using PowerShell.
由于列数很大,因此用户界面处理它们的速度变得非常慢,因此我想切换到使用PowerShell。
I put together a script that I am able to use to update them from a JSON file using the Set-AzureRmDataFactoryV2Dataset
and Set-AzureRmDataFactoryV2Pipeline
cmdlets. 我整理了一个脚本,可以使用
Set-AzureRmDataFactoryV2Dataset
和Set-AzureRmDataFactoryV2Pipeline
cmdlet从JSON文件更新它们。 I'd like to also be able to generate JSON files with the current configurations using PowerShell. 我还希望能够使用PowerShell使用当前配置生成JSON文件。 I've tried using the matching
Get-AzureRmDataFactoryV2Dataset
and Get-AzureRmDataFactoryV2Pipeline
cmdlets which have been useful for getting general information about them, but I haven't been able to go from the PSDataSet object returned to a valid JSON file. 我尝试使用匹配的
Get-AzureRmDataFactoryV2Dataset
和Get-AzureRmDataFactoryV2Pipeline
cmdlet,这些cmdlet对于获取有关它们的常规信息很有用,但是我无法从PSDataSet对象返回到有效的JSON文件。 Below are the commands that I've been trying, but it ends up giving me a bunch of empty JSON arrays for the schema. 以下是我一直在尝试的命令,但最终给了我一堆用于该模式的空JSON数组。
$dtSrc = Get-AzureRmDataFactoryV2Dataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $dtSrcName
ConvertTo-Json $dtSrc.Properties -Depth 100 | Out-File "$dtSrcName.json" -Width 1000000
Edit: Based on the answer by Wang Zhang, I've edited my ConvertTo-Json
statement as shown below. 编辑:根据Wang Zhang的回答,我编辑了
ConvertTo-Json
语句,如下所示。 The output looks much better now (no empty arrays), but it still doesn't match the Azure Dataset JSON schema. 现在,输出看起来好得多(没有空数组),但是仍然与Azure数据集JSON架构不匹配。
$dtSrc = Get-AzureRmDataFactoryV2Dataset -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $dtSrcName
ConvertTo-Json $dtSrc -Depth 1 | Out-File "$dtSrcName.json" -Width 1000000
would you please tell the reason that you set " -Depth -100 " in the second Command? 您能否告诉您在第二个命令中设置“ -Depth -100 ”的原因? Seems remove it and modify the second command as the following works well:
似乎将其删除并修改第二条命令,因为以下操作效果良好:
ConvertTo-Json $dtSrc.Properties | ConvertTo-Json $ dtSrc.Properties | Out-File "$dtSrcName.json" -Width 1000000
输出文件“ $ dtSrcName.json”-宽度1000000
Please have a try. 请尝试一下。
You could just download the ARM templates for your ADF (they use the json format) edit them and upload them again, either by using the UI, az-cli or even powershell. 您可以下载ADF的ARM模板(它们使用json格式),然后使用UI,az-cli甚至是powershell对其进行编辑并再次上传。
Or even better, link a git repository to the ADF and then just clone the repo locally, make your changes and publish them to ADF by doing a push to the specified production branch (normally master). 甚至更好的方法是,将git存储库链接到ADF,然后只需在本地克隆存储库,进行更改并将其推送到指定的生产分支(通常是主分支)将它们发布到ADF。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.