使用htmlagility的C＃抓取网址

Question

Okay so I have this list of URLs on this webpage, I am wondering how do I grab the URLs and add them to a ArrayList?

http://www.animenewsnetwork.com/encyclopedia/anime.php?list=A

I only want the URLs which are in the list, look at it to see what I mean. I tried doing it myself and for whatever reason it takes all of the other URLs except for the ones I need.

   http://pastebin.com/a7hJnXPP

Answer 1

Using Html Agility Pack

using (var wc = new WebClient())
{
    HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    doc.LoadHtml(wc.DownloadString("http://www.animenewsnetwork.com/encyclopedia/anime.php?list=A"));
    var links = doc.DocumentNode.SelectSingleNode("//div[@class='lst']")
        .Descendants("a")
        .Select(x => x.Attributes["href"].Value)
        .ToArray();
}

Answer 2

If you want only the ones in the list, then the following code should work (this is assuming you have the page loaded into an HtmlDocument already)

List<string> hrefList = new List<string>(); //Make a list cause lists are cool.

foreach (HtmlNode node animePage.DocumentNode.SelectNodes("//a[contains(@href, 'id=')]"))
{
    //Append animenewsnetwork.com to the beginning of the href value and add it
    // to the list.
    hrefList.Add("http://www.animenewsnetwork.com" + node.GetAttributeValue("href", "null"));
}

//a[contains(@href, 'id=')] Breaking this XPath down as follows:

• //a Select all <a> nodes...
• [contains(@href, 'id=')] ... that contain an href attribute that contains the text id= .

That should be enough to get you going.

As an aside, I would suggest not listing each link in its own messagebox considering there are around 500 links on that page. 500 links = 500 messageboxes :(