我有一个网站链接，如何从该网站下载所有文件？

Question

For example: http://www.test.com My program is digging crawling. So i want that it will download all the files each time.

For example:

using (WebClient Client = new WebClient ())
{
    Client.DownloadFile("http://www.abc.com/file/song/a.mpeg", "a.mpeg");
}

This will download only the specific a.mpeg file. I want to do something like:

using (WebClient Client = new WebClient ())
{
    Client.DownloadFile(address, "*.*");
}

Since the address is changing all the time and i want to download all the files not a specific file like mpeg or jpg or avi...any extetion.

Doing " . " is the right way ?

EDIT**

This is how i'm downloading images today:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using HtmlAgilityPack;
using System.IO;
using System.Text.RegularExpressions;
using System.Xml.Linq;
using System.Net;
using System.Web;
using System.Threading;
using DannyGeneral;
using GatherLinks;

namespace GatherLinks
{
    class RetrieveWebContent
    {
        HtmlAgilityPack.HtmlDocument doc;
        string imgg;
        int images;

        public RetrieveWebContent()
        {
            images = 0;
        }

        public List<string> retrieveFiles(string address)
        {

        }

        public List<string> retrieveImages(string address)
        {

            System.Net.WebClient wc = new System.Net.WebClient();
            List<string> imgList = new List<string>();
            try
            {
                    doc = new HtmlAgilityPack.HtmlDocument();
                    doc.Load(wc.OpenRead(address));
                    string t = doc.DocumentNode.InnerText;
                    HtmlNodeCollection imgs = doc.DocumentNode.SelectNodes("//img[@src]");
                    if (imgs == null) return new List<string>();

                    foreach (HtmlNode img in imgs)
                    {
                        if (img.Attributes["src"] == null)
                            continue;
                        HtmlAttribute src = img.Attributes["src"];
                        imgList.Add(src.Value);
                        if (src.Value.StartsWith("http") || src.Value.StartsWith("https") || src.Value.StartsWith("www"))
                        {
                            images++;
                            string[] arr = src.Value.Split('/');
                            imgg = arr[arr.Length - 1];
                            //imgg = Path.GetFileName(new Uri(src.Value).LocalPath);
                            //wc.DownloadFile(src.Value, @"d:\MyImages\" + imgg);
                            wc.DownloadFile(src.Value, "d:\\MyImages\\" + Guid.NewGuid() + ".jpg");
                        }
                    }
                return imgList;
            }
            catch
            {
                Logger.Write("There Was Problem Downloading The Image: " + imgg);
                return null;

            }
        }
    }
}

Now in this place of code:

public List<string> retrieveFiles(string address)
        {

        }

I dont want to download only jpg files but any type of files. And if the link is for example: http://tes.com \\i.jpg why i need to parse the website instead of save as it somehow ?

Answer 1

No, WebClient.DownloadFile will never ever act like a Crawler. You would need to download the page and use a C# HtmlParser on the returned page HTML, enumerate over the resources you are interested in and download them all individually.