DownloadFileTaskAsync 无法下载数千个文件

Question

I have a thousands list of download url. The url may contain a different type of file such as that image, pdf, audio, video etc. I am trying to download them using DownloadFileTaskAsync. But after several files downloading the app is breaking down. I don't understand what is happening. don't get any error message. Just my app is shut down after few files downloading.

foreach (var url in urls)
{
    //if file exists in our local directory then do not need to download and continue the process...
    if (FileExistsOrNot(localPath + "/" + url.fileName))
       continue;

    Thread thread = new Thread(async () => {
        WebClient client = new WebClient();
        client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(client_DownloadProgressChanged);
        client.DownloadFileCompleted += new AsyncCompletedEventHandler(client_DownloadFileCompleted);
        await client.DownloadFileTaskAsync(new Uri(url.downloadUrl), localPath + "/" + url.fileName);  
    });
    thread.Start();
}

Answer 1

Spawning a new thread for every download it's no wonder that you quickly run out of system resources (threads in this case).

You should use some kind of thread pooling and luckily, C# has multiple out-of-the-box solution to do this:

• ThreadPool - ThreadPool.QueueUserWorkItem
• Parallel Programming - Parallel.ForEach
• Tasks + TaskScheduler - Task.WhenAll

ThreadPool is the lowest level construct, parallel programming and task schedulers are built on top of that.

As you use an async operation to download ( DownloadFileTaskAsync ), tasks are the most suitable option in this case:

Action<string, DownloadProgressChangedEventArgs> onDownloadProgress = (url, e) => 
{ 
    /* your logic displaying progress... */ 
};

var downloadTasks = urls
    .Where(url => !FileExistsOrNot(Path.Combine(localPath, url.fileName)))
    .Select(async url =>
    {
        using (var client = new WebClient())
        {
            client.DownloadProgressChanged += (s, e) => onDownloadProgress(url.fileName, e);
            try { await client.DownloadFileTaskAsync(new Uri(url.downloadUrl), Path.Combine(localPath, url.fileName)); }
            catch (Exception ex) { /* handle download error: log exception, etc */ }                
        }
    });    

Task.WaitAll(downloadTasks.ToArray()); // or Task.WhenAll(...) if you want it non-blocking

Some remarks:

• use Path.Combine to build path strings to get correct results on every platform
• WebClient implements IDisposable , that is, it may use unmanaged resources. You should call Dispose on it as soon as you don't need it anymore, otherwise you can run out of system resources again.
• you don't need to observe DownloadFileCompleted . Awaiting DownloadFileTaskAsync rethrows potential errors as exceptions.
• consider using HttpClient instead of WebClient for better async support and performance