基于javascript的网站有何不同？

Question

I am trying to scrape the content of a website which seems to be working on javascript or some other technology. I am using xpath to find the content on the page. I can see the content using firebug in the browser but if i save the source or download the source code via curl/wget, content is missing from the page. How is this possible ?

thanks in advance

Answer 1

Some content are loaded via JS dynamically. You need to run the JS somehow, like in a headless browser like PhantomJS for several seconds in order to load dynamic content. Then run through the DOM, similar to how .html() in jQuery does it, to get the rendered content.

As far as I know, this is similar to how Opera Mini does it in their proxies before they re-encode and send it to your device:

The server sends the response back as normal — when this is received by the Opera transcoding servers, they parse the markup and styles, execute the JavaScript, and transcode the data into Opera Binary Markup Language (OBML). This OBML data is progressively loaded by Opera Mini on the user's device.

Opera Mini's entry from Wikipedia :

JavaScript will only run for a couple of seconds on the Mini server before pausing, due to resource constraints.

According to the documentation for Opera Mini 4, before the page is sent to the mobile device, its onLoad events are fired and all scripts are allowed a maximum of two seconds to execute. The setInterval and setTimeout functions are disabled, so scripts designed to wait a certain amount of time before executing will not execute at all. After the scripts have finished or the timeout is reached, all scripts are stopped and the page is compressed and sent to the mobile device.

Answer 2

Typically the page loads and then requests the content (ajax) which is returned as json or jsonp. This is usually pretty handy for scraping because json is even easier to parse than html.

But if you haven't done it before, it can be a challenge to figure out how to make the right ajax request.