How to Read a file from Azure Data Lake Storage with file Url?

Question

Is there a way to read files from the Azure Data Lake. I have the Http url of the file. I want to read it direclty. How can i acheive it because I don't see a way to do it via the SDK.

Thanks for your help.

Regards

Answer 1

Did you check docs ?

public async Task ListFilesInDirectory(DataLakeFileSystemClient fileSystemClient)
{
    IAsyncEnumerator<PathItem> enumerator = 
        fileSystemClient.GetPathsAsync("my-directory").GetAsyncEnumerator();

    await enumerator.MoveNextAsync();

    PathItem item = enumerator.Current;

    while (item != null)
    {
        Console.WriteLine(item.Name);

        if (!await enumerator.MoveNextAsync())
        {
            break;
        }

        item = enumerator.Current;
    }

}

You can also use ADLS Gen2 rest api ,

For example, you can write code like below with sas token authentication(or you can also use the shared key authentication):

    string sasToken = "?sv=2018-03-28&ss=b&srt=sco&sp=rwdl&st=2019-04-15T08%3A07%3A49Z&se=2019-04-16T08%3A07%3A49Z&sig=xxxx";
    string url = "https://xxxx.dfs.core.windows.net/myfilesys1/app.JPG" + sasToken;
    var req = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));

    //you can specify the Method as per your operation as per the api doc
    req.Method = "HEAD"; 
    var res = (HttpWebResponse)req.GetResponse();

Answer 2

If you know Blob APIs and Data Lake Storage Gen2 APIs can operate on the same data , then you can directly use the azure blob storage SDK to read file from ADLS Gen2.

First, install this nuget package: Microsoft.Azure.Storage.Blob, version 11.1.6 .

Note that , in this case, you should use this kind of url "https://xxx.blob.core.windows.net/mycontainer/myfolder/test.txt " instead of that kind of url " https://xxx.dfs.core.windows.net/mycontainer/myfolder/test.txt ".

Here is the sample code which is used to read a.txt file in ADLS Gen2:

        var blob_url = "https://xxx.blob.core.windows.net/mycontainer/myfolder/test.txt";
        //var blob_url = "https://xxx.dfs.core.windows.net/mycontainer/myfolder/test.txt";

        var username = "xxxx";
        var password = "xxxx";

        StorageCredentials credentials = new StorageCredentials(username, password);
        var blob = new CloudBlockBlob(new Uri(blob_url),credentials);
        var mystream = blob.OpenRead();
        using (StreamReader reader = new StreamReader(mystream))
        {
            Console.WriteLine("Read file content: " + reader.ReadToEnd());
        }


        //you can also use other method like below
        //string text = blob.DownloadText();
        //Console.WriteLine($"the text is: {text}");

The test result: