HttpwebRequest模拟点击

Question

I was working on httpwebrequest and was trying to search google get result and simulate click to desired link. Is that possible?

 string raw ="http://www.google.com/search?hl=en&q={0}&aq=f&oq=&aqi=n1g10";
string search = string.Format(raw, HttpUtility.UrlEncode(searchTerm));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(search);
request.Proxy = prox;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.ASCII))
{
HtmlElementCollection html = reader.ReadToEnd();
browserA=reader.ReadToEnd();
this.Invoke(new EventHandler(IE1));
}
}

Answer 1

You could parse the page using http://htmlagilitypack.codeplex.com/ or http://www.justagile.com/linq-to-html.aspx (also you may use Regexps if needed in conjunction with this tools) to find elements you want to "Click" and then process HttpWebRequest with this new elements. It is calling http://en.wikipedia.org/wiki/Web_scraping .

Also you should remember that resource which you web scraping may ban your IP address if a lot of requests coming from your IP address, to avoid that you need to think about using list of proxy servers.

Answer 2

A better option is to use one of google's APIs.

There is a list of all of them here: Google APIs

Here is another on codeplex: Google Dot Net

They have services that allow applications to use google freely. With most of these there are wsdl files you can use to "Add Web Reference" in Visual Studio.

Using Regex and HtmlAgility pack should only be used as a last resort when a website does not expose public services (I had to use it recently for something I'm writing to integrate to uTorrent and BtJunkie). Google obviously wants people to develop with their sites in these ways.