自动将 .whl 工件从 Azure DevOps 源部署到 Synapse Spark 池

Question

I want to automatically deploy artifacts (custom python packages / .whl files) from the Azure DevOps artifact feed to a Synapse spark pool.

Currently I have to manually:

• download the .whl file from the artifact feed
• upload the .whl file to the synapse workspace
• add the package/.whl to the packages of a spark pool (> "Select from workspace packages").

I have so far not found any option to do this as part of a release/pipeline in Azure DevOps or via the AzureCLI and also have not found any documentation on it. I was wondering whether anyone has found a solution concerning automating this step? It is quite cumbersome to have to do it manually.

PS: I already asked this question on the MS forum , but have gotten no official answer yet.

Answer 1

I managed to update the pool spark packages by using both the Download Packages task and the Azure CLI task in the release pipeline on devops.

First use the Download Packages task to download your feed to the pipeline $(System.DefaultWorkingDirectory) then use the Azure CLI to update the sparkpool.

In the Azure CLI task write in the inline command field the two following lines. Use the $(System.DefaultWorkingDirectory) as working directory, also I recommand to check the Use global Azure CLI configuration checkbox.

• First command to upload your .whl to the workspace. Here is the doc

  az synapse workspace-package upload --workspace-name yourworkspace --package yourpackage.whl

• Then upload it to the pool spark. Here is the doc .

  az synapse spark pool update --name yourpoolspark --workspace-name yourworkspace --resource-group yourresourcegroupe --package-action Add --package yourpackage.whl

Those two commands will add the package to your workspace and then upload it to the poolspark.

Hope it will help you !