How to use Jenkins to upload to Azure Data Lake Storage Gen1?

Question

I found the Windows Azure Storage Plugin for Jenkins. But it seems that it works only with blob storage. Is there a way to upload files to Azure Data Lake Storage Gen1 from Jenkins ? Thanks for your help !

Answer 1

AFAIK currently there is no Jenkins plugin to upload files to Azure Data Lake Storae Gen1 from Jenkins. You could use Azure PowerShell or Azure CLI to accomplish your requirement.

Sample Azure PowerShell command:

Import-AzDataLakeStoreItem -AccountName $dataLakeStorageGen1Name -Path "C:\sampledata\vehicle1_09142014.csv" -Destination $myrootdir\mynewdirectory\vehicle1_09142014.csv

Sample Azure CLI command:

az dls fs upload --account mydatalakestoragegen1 --source-path "C:\SampleData\AmbulanceData\vehicle1_09142014.csv" --destination-path "/mynewfolder/vehicle1_09142014.csv"

For more information, refer below articles:

https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-powershell#upload-data-to-your-data-lake-storage-gen1-account

https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-cli-2.0

https://docs.microsoft.com/en-us/powershell/module/az.datalakestore/import-azdatalakestoreitem?view=azps-1.5.0

https://docs.microsoft.com/en-us/cli/azure/dls/fs?view=azure-cli-latest#az-dls-fs-upload

Few pre-requisites to make this work without any issues are:

1. Have Azure PowerShell / Azure CLI installed in the node where you run the command(s).

2. Add Azure service principal to Jenkins credential as instructed here -> https://docs.microsoft.com/en-us/azure/jenkins/execute-cli-jenkins-pipeline#add-azure-service-principal-to-jenkins-credential

Hope this helps!! Cheers!!