javascript生成的python goto链接

Question

everybody! i need to get some data from website, that generate its content with javascript. somehow, with ClientForm module help, i could authorize on the site and get needed data by passing needed form values to search lists. but than, i get a few pages of results. and i need to loop over that pages, saving them. but the point is that 'NEXT' page button looked like

a onclick="if(typeof jsfcljs == 'function') { 
               jsfcljs(document.getElementById('nestedView:j_id_id8pc10'),           
                   {'nestedView:j_id_id8pc10:next':'nestedView:j_id_id8pc10:next'},
                   '');
           }
           return false"
href="#" id="nestedView:j_id_id8pc10:next">next

how could i move on this link with python?

Answer 1

Your options are essentially:

1. Work out what the javascript is doing, and try to copy it in Python, so that the server sends you the same data. If you're not already using it, Firebug is the sort of thing you'll need.
2. Use a browser control framework, such as Selenium RC, to interact with a browser using Python.

Generally, option 1 is more straightforward unless you're already familiar with using a framework to control a browser, or the javascript is really, horribly hard to follow.

Oh, and 3: Write a javascript+DOM engine in Python. (No, that's a joke. There's a reason one doesn't already exist.)

Answer 2

maybe i can simulate a POST request (capture it from Live HTTP Headers in firefox, for example) and navigate on site such way? cause the main point is to get data...and steps to get it are similar (log in, fill 2 search forms, loop through results pages)