Azure data lake connection with data factory custom activity

Question

I have a problem with custom activity in azure data factory. I want to connect to azure data lake within it. I am using following namespaces:

using Microsoft.Azure.Management.DataLake.Store;
using Microsoft.IdentityModel.Clients.ActiveDirectory
using Microsoft.Rest.Azure.Authentication;

And here is my code:

    public ConnectDataLakeStore(string applicationId, string applicationSecretKey, string tenantId)
    {
        var credentials = new ClientCredential(applicationId, applicationSecretKey);
        var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, credentials).Result;
        var fileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);
    }

And I get the following exception:

Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Microsoft.Azure.Management.DataLake.Store.Models.AdlsErrorException: Operation returned an invalid status code 'Forbidden'
   at Microsoft.Azure.Management.DataLake.Store.FileSystemOperations.<ListFileStatusWithHttpMessagesAsync>d__12.MoveNext()

When I run custom activity in local environment everything works fine. Problem occurs after deployment to azure. I also gave all possible permissions to ad application.

Any hints?

Answer 1

I gather from your comment above that you've already solved this. But still thought it worth adding a little more detail for future peeps with similar issues.

First create a credential using your service principal and domain details.

Method:

    private static ServiceClientCredentials AuthenticateAzure(string domainName, string clientID, string clientSecret)
    {
        SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());

        var clientCredential = new ClientCredential(clientID, clientSecret);
        return ApplicationTokenProvider.LoginSilentAsync(domainName, clientCredential).Result;
    }

Maybe get the values from your library config or project properties.

var creds = AuthenticateAzure(domainName, appId, appPass);

Next create an instance of the data lake account manager or file system manager depending on what you need.

    private static DataLakeStoreFileSystemManagementClient adlsFileSystemClient;
    private static DataLakeStoreAccountManagementClient adlsAccountManagementClient;

    adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds);
    adlsAccountManagementClient = new DataLakeStoreAccountManagementClient(creds);

Next create something like the below method to upload files to the data lake store.

The uploader will perform faster than the create file method.

    private static void UploadFile(string srcFilePath, string destFilePath, string accName, bool force = true)
    {
        var parameters = new UploadParameters(srcFilePath, destFilePath, accName, isOverwrite: force);
        var frontend = new DataLakeStoreFrontEndAdapter(accName, adlsFileSystemClient);
        var uploader = new DataLakeStoreUploader(parameters, frontend);
        uploader.Execute();
    }

Sources:

ADF Custom Activities https://www.purplefrogsystems.com/paul/2016/11/creating-azure-data-factory-custom-activities/

ADL Auth https://www.purplefrogsystems.com/paul/2016/12/azure-data-lake-authentication-from-azure-data-factory/

Hope this helps