用Node.js进行爬取

Question

I have a strange problem-

Because of the strange way that this site presents time data, I wanted to write a small parser.

I was testing my code on one specific url -

http://www.sfweekly.com/search/results/?keyword=*&type=events#type:events/page:57/

Note that when you visit the url, the page first loads a bunch of entries and then changes those entries. What's happening there is that it's going to the first page and then re-directing. How do I get around that?

To scrape I'm using

jsdom.env({
    html: url,
    scripts:['http://code.jquery.com/jquery.js'],
    done: function(errors,window){
                 //doSomething

I originally thought I could get around this with a pause, but that's not the case. Is there some way I can 'listen' for a redirect and wait until the real page has been loaded? I also have a feeling that the new entries may be entered with a jquery replace, but I'm not sure how to test that theory.

Answer 1

Scraping ajax-y sites like this can be a real pain. In this case, it seems there is a way around it, because you can snoop around in the developer tools in your browser of choice and discover the ajax endpoint, and use that directly:

http://www.sfweekly.com/search/ajaxsearch/type%3aevents/page:57/

In some scenarios, javascript-y sites that intentionally try to foil scrapers, you have to use some kind of headless or automated browser situation. That's slow and annoying, avoid it if you can.