搜索结果后，屏幕抓取/网络抓取网址不变

Question

I'm working on a web scraping project and I need my "start" page to start the scraping.

When navigating to a URL like this: http://www.blah.com/search.aspx and I enter in my data (because I want to pass the search results page to my screen scraping program). When the search results (page) comes back, the URL did not change.

How can I get the URL from the search results page, or, mimic the "search" to get the search results page in the screen scraping program?

Answer 1

Found this after some more googling (it helps to use the right keywords).

I was already using HtmlAgilityPack, this just provides more insight on how to initiate the post and read the results.

http://refactoringaspnet.blogspot.com/2010/04/using-htmlagilitypack-to-get-and-post.html