如何为有关人员的新闻文章构建.NET Web scraper

Question

I am looking to create a simple webservice to crawl webpages on specific websites and look for a person's name. Anybody know if there are any examples out there of this, or can anyone help me with the start of this?

Edit: I should mention I want to do this with Visual Studio C#. I will only be looking at English news sites that I specify.

Answer 1

Here is a simple function that returns true if a Web page contains a person's name:

string response;
using (System.Net.WebClient wc = new System.Net.WebClient())
{
    response = wc.DownloadString(url);
}  
return reponse.Contains("John Doe");

For finding the links within the page, check out this question: Parse HTML links using C#
You can collect distinct Urls throughout the site and run the code above for each Url you find.

Also, type this into Google to see what they find. site:www.somesite.com "John Doe"

Answer 2

Using c# your best option for a crawler and parser (the two parts to your solution) would be to use functionality exposed by the HtmlAgility Pack, which can be found on CodePlex.

Refer to this answer for an example usage scenario: How to use HTML Agility pack