Download multiple files in parallel using c#

Question

I want to download files in parallel using C#. For this, I have written this code which is working perfectly but the problem is that UI is freezing.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using System.Windows.Threading;

namespace FileDownloader
{
    /// <summary>
    /// Interaction logic for MainWindow.xaml
    /// </summary>
    public partial class MainWindow : Window
    {
        private static int count = 1;
        private static string f= "lecture";
        private string URL = "www.someexample.com";

        public MainWindow()
        {
            InitializeComponent();
        }

        public static string GetDirectoryListingRegexForUrl(string url)
        {
            if (url.Equals(URL))
            {
                return "<a href=\".*\">(?<name>.*)</a>";
            }
            throw new NotSupportedException();
        }

        public  void DownloadP(string[] urls)
        {
            Parallel.ForEach(urls.ToList(), new ParallelOptions { MaxDegreeOfParallelism = 10 }, DownloadFile);
        }

        private void DownloadFile(string url)
        {
           using(WebClient client=new WebClient())
           {
               if (url.EndsWith(".pdf"))
               {
                   int nextIndex = Interlocked.Increment(ref count);

                   client.DownloadFile(url, f + nextIndex + ".pdf");
                   this.Dispatcher.Invoke(() => {
                       listbox.Items.Add(url);
                   });
               }
           }
        }

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            DownloadP(listofFiles);
        }
    }
}

Answer 1

You can use async/await with conjunction with new WebClient method DownloadFileTaskAsync .

private async Task DownloadFile(string url)
{
    if (!url.EndsWith(".pdf"))
    {
        return;
    }

    using (var client = new WebClient())
    {
        int nextIndex = Interlocked.Increment(ref count);

        await client.DownloadFileTaskAsync(url, "lecture" + nextIndex + ".pdf");
        listBox.Items.Add(url);

    }
}

private async void Button_OnClick(object sender, RoutedEventArgs e)
{
    button.IsEnabled = false;
    await DownloadFiles(urlList);
    button.IsEnabled = true;
}

private async Task DownloadFiles(IEnumerable<string> urlList)
{
    foreach (var url in urlList)
    {
        await DownloadFile(url);
    }
}

Answer 2

instead of using client.DownloadFile use client.DownloadFileAsync like this

var webClient=new WebClient();
webClient.DownloadFileCompleted += webClient_DownloadFileCompleted;
webClient.DownloadFileAsync("Your url","file_name");

the event

    private void webClient_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
    {
        MessageBox.Show("Download Completed.");
    }

Answer 3

I know this Question is old, but WebClient is outdated and dedicated. First of all, WPF uses MVVM pattern that you should try to follow, but except that you can use the HttpClient in using System.Net.Http; .

To download multiple files in a parallel you can create a parallel foreach that handles all the downloads that the HttpClient have to perform. You did tis right but the foreach will block the thread so start it in a new one Task.Run(()=>{// Your Code});

In the case you don't want do write something like that yourself, you could use the Shard Download Library on NuGet. This NuGet Package is also on GitHub if you want to see how it does its job. It helped me often if I want do download a lot of files, because is does not block the UI thread and is easy to use. To use it you have to write something like this:

void Download()
{
    string[] links = new string[]{ 
        "https://google.com",
        "https://speed.hetzner.de/100MB.bin",
        "https://file-examples.com/storage/fe88dacf086398d1c98749c/2017/04/file_example_MP4_1920_18MG.mp4" };
    foreach (var link in links)
    {
        _ = new LoadRequest(link);
    }
    Console.ReadLine();
}

But you can also set the MaxDegreeOfParalism and you can change it while it is downloading. You can set the Path for the output file and name it. The problem that the HttpClient has with download finished and progress report is also solved with this. The bad thing is that the documentation of this library is not very good. Here an example with few options as an example.

async Task DownloadAsync()
{
    string[] links = new string[]{
        "https://google.com",
        "https://speed.hetzner.de/100MB.bin",
        "https://file-examples.com/storage/fe88dacf086398d1c98749c/2017/04/file_example_MP4_1920_18MG.mp4" };
    RequestHandler requestHandler = new()
    {
        MaxDegreeOfParallelism = 10
    };

    LoadRequestOptions option = new()
    {
        Progress = new Progress<float>(value => Console.WriteLine(value.ToString("0.0%"))),
        DestinationPath = "C:\\Users\\[UserName]\\Desktop\\",
        RequestCompleated = path => Console.WriteLine("Finished: " + path?.ToString()),
        RequestHandler = requestHandler
    };
    LoadRequest last = null;
    foreach (string link in links)
    {
        last = new LoadRequest(link, option);
    }
    await last.Task;
} 

I hope that can help people that have the same problem and don't want to use the dedicated WebClient.

Answer 4

Relace your DownloadP function with this :

public async Task DownloadP(string[] urls)
{
  await Task.Factory.StartNew(() => Parallel.ForEach(urls.ToList(), new ParallelOptions { MaxDegreeOfParallelism = 10 }, DownloadFile));
}