从Azure Data Lake下载文件

Question

I upload my files in azure data lake. I try to download that file through asp.net mvc application.I have adl path for that file. I can download below 150 MB files. But i can't download the more then 150 MB files. Time out error came.

My Code in the bellow...

public ActionResult Download(string adlpath)
{
    String header = adlpath;
    Console.WriteLine(header);
    string[] splitedStr = header.Split('/');
    var path = GenerateDownloadPaths(adlpath);
    string filename = path["fileName"];
    HttpResponseMessage val = DataDownloadFile(path["fileSrcPath"]);
    byte[] filedata = val.Content.ReadAsByteArrayAsync().Result;
    string contentType = MimeMapping.GetMimeMapping(filename);
    var cd = new System.Net.Mime.ContentDisposition
    {
        FileName = filename,
        Inline = true,
    };
    Response.AppendHeader("Content-Disposition", cd.ToString());

    return File(filedata, contentType);
}

public HttpResponseMessage DataDownloadFile(string srcFilePath)
{
    string DownloadUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?op=OPEN&read=true";
    var fullurl = string.Format(DownloadUrl, _datalakeAccountName, srcFilePath);

    using (var client = new HttpClient())
    {
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _accesstoken.access_token);
        using (var formData = new MultipartFormDataContent())
        {
            resp = client.GetAsync(fullurl).Result;
        }
    }
    return resp;
}

Image :

Answer 1

You should modify your code to use async and await . Your implementation blocks while retrieving the file and that is probably what times out:

public async Task<HttpResponseMessage> DataDownloadFile(string srcFilePath)
{
    string DownloadUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?op=OPEN&read=true";
    var fullurl = string.Format(DownloadUrl, _datalakeAccountName, srcFilePath);

    using (var client = new HttpClient())
    {
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _accesstoken.access_token);
        using (var formData = new MultipartFormDataContent())
        {
            resp = await client.GetAsync(fullurl);
        }
    }
    return resp;
}

The return value of the method is changed to Task<HttpResponseMessage> and the async modifier is added.

Calling client.GetAsync is changed to use await instead of blocking by retrieving the Result property.

Your code may still timeout. I believe that there is a configurable limit on how long a request can take before it is aborted and if you still get a timeout you should investigate this.

Answer 2

Per my understanding, you could try to increase the HttpClient.Timeout (100 seconds by default) for your HttpClient instance.

HttpClient.Timeout

Gets or sets the timespan to wait before the request times out.

The default value is 100,000 milliseconds (100 seconds).

Moreover, if you host your application via Azure Web App, you may encounter an idle timeout setting of 4 minutes from Azure Load Balancer. You could change the idle timeout setting in Azure VM and Azure Cloud Service. Details you could follow here .