How to scrape a page generated with a script in C#?

Question

Simple example: Google search page.

http://www.google.com/search?q=foobar

When I get the source of the page, I get the underlying JavaScript. I want the resulting page. What do I do?

Answer 1

Even though it looks as if it is only javascript it really is the full HTML, you can easily confirm with HtmlAgilityPack :

HtmlAgilityPack.HtmlWeb web = new HtmlAgilityPack.HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = web.Load("http://www.google.com/search?q=foobar");
string html = doc.DocumentNode.OuterHtml;
var nodes = doc.DocumentNode.SelectNodes("//div"); //returns 85 nodes