Web scraping a page with Javascript using Python

Question

I am trying to scrape a web page that is a list of urls using Python. I can scrape the first page with no problem using Python and beautiful soup, however if the the url list is long it continues on a second page using the following JavaScript.

href="javascript:__doPostBack('WQResultGridView','Page$2')

I don't know how to get to the second page.

Answer 1

I guess this call would be translated into an http (post, as the name suggests) request.

look at firebug's 'net' tab for how that request is structured and run the same request using python.

another, more generic way to handle this would be to load the page and run the javascript it holds using a tool like http://phantomjs.org/