Read CSV From Azure Data lake storage Gen 1 in c# .net API

Question

We need to read a CSV File of around 2 GB which is stored in Azure Data lake storage Gen1 .The purpose is like we have to render the data in Grid format (UI ) with high performance when user request. We are using .Net Core 2.1 (c#) for doing API for the same.

        var creds = new ClientCredential(applicationId, clientSecret);
        var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();

        // Create ADLS client object
        AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);

        string fileName = "/cchbc/sources/MVP/Data.csv";

        using (var readStream = new StreamReader(client.GetReadStream(fileName)))
        {

            while ((line = readStream.ReadLine()) != null)
            {
                content = content + line;
            }
        }

I have tried the above code but failed with an error GETFILESTATUS failed with HttpStatus:Forbidden RemoteException: AccessControlException GETFILESTATUS failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.)

Any suggestion will be very beneficial.Thanks in advance

Answer 1

If you want to use a service principal to access files storing in Azure data lake gen 1, we need to configure ACL for the service principal. The ACL has three permissions Read(read the contents of a file) Write(write or append to a file) and Execute(traverse the child items of a folder).

for example I access file /test/test.csv

1. Configure ACL as below

Opreation    Object        /        test/      test.csv
Read         tets.csv      --X      --X        R-- 

2. Code

string appId = "service principal appId";
            string appSecret = "service principal appSecret";
            string domain = "service principal domain";

            var serviceSettings = ActiveDirectoryServiceSettings.Azure;
            serviceSettings.TokenAudience = new Uri(@"https://datalake.azure.net/");
            
            var creds = await ApplicationTokenProvider.LoginSilentAsync(domain, appId, appSecret, serviceSettings);

            string accountName = "testadls02";
            AdlsClient client = AdlsClient.CreateClient($"{accountName}.azuredatalakestore.net", creds);
            string fileName = "/test/test.csv";
            string line = null;
            using (var readStream = new StreamReader(client.GetReadStream(fileName)))
            {
                while ((line =  await readStream.ReadLineAsync()) != null) {

                    Console.WriteLine(line);
                }
               
            }

For more details, please refer to here