同步下载多个文件C＃

Question

I'm trying to download multiple files with several threads. The program uses a BFS algorithm to reach all the files given a particular url: http://www.police.am/Hanraqve/ The problem is that the same file can be downloaded multiple times as several threads are released. I'm thinking of a way to synchronize the download process so that each file is downloaded once only with the help of Mutexes or Semaphores. Any idea or actual code would be very much appreciated. Here is my initial code

    public static async Task Download()
    {
        nodes.Enqueue(root);
        while (nodes.Count() != 0)
        {
            String currentNode = "";
            if (nodes.TryDequeue(out currentNode))
            {
                if (!visitedNodes.Contains(currentNode))
                {
                    visitedNodes.Add(currentNode);
                    if (isFolder(currentNode))
                    {
                        List<String> urls = GetUrlsFromHtml(currentNode);
                        foreach (String url in urls)
                        {
                            nodes.Enqueue(url);
                        }
                    }
                    else
                    {
                        string fileName = currentNode.Remove(0, currentNode.LastIndexOf('/') + 1);

                        using (WebClient webClient = new WebClient())
                        {
                            await webClient.DownloadFileTaskAsync(new Uri(currentNode), destinationFolderPath + @"\" + fileName);
                            files.Enqueue(destinationFolderPath + @"\" + fileName);
                        }
                    }
                }

            }
        }


        //cts.Cancel();
    }

    public static List<String> GetUrlsFromHtml(string url)
    {
        HtmlWeb hw = new HtmlWeb();
        HtmlDocument doc = hw.Load(url);
        List<String> urls = new List<String>();
        foreach (HtmlNode htmlNode in doc.DocumentNode.SelectNodes("//a[@href]"))
        {
            string hrefValue = htmlNode.Attributes["href"].Value;
            if (hrefValue[0] >= '1' && hrefValue[0] <= '9')
            {
                urls.Add(url + hrefValue);
            }
        }
        return urls;
    }

    public static bool isFolder(string url)
    {
        return url.EndsWith("/");
    }
}

}

Answer 1

Check the urls your storing in visited, they may be different but still go to the same page.

http://foo.com?a=bar http://foo.bar?b=foo