如何在该网站上抓取动态生成的数据？

Question

Here is the website I want to scrape http://www.quickbid.com.tw/

I wish I can get class="timestamp" into a variable in python so that I can parse "timestamp" in the way I like.

I've tried using scrapy to scrape "timestamp", but because scrapy does not support javascript-generated data, I cannot get it.

I also tried using firebug to monitor the packets transmitted between "quickbid" and my browser. I found that there are packets being transmitted every second in order to synchronize the timestamp. But I still don't know how these packets are generated. I heard that maybe Selenium can help me reach my goal. But after reading the tutorials of Selenium( http://www.jroller.com/selenium/ ), I still can't get clues of how to scrape the data I want.

Does anyone know how to scrape data from this website? Any help will be appreciated.

Answer 1

I usually use basic requests and BeautifulSoup libraries for scrapping. I did this:

import requests
from bs4 import BeautifulSoup
r = requests.get("http://www.quickbid.com.tw/")
c = r.content
soup = BeautifulSoup(c,'html.parser')
timestanp = soup.findAll('span',{'class':'timestamp'})
print timestanp

it returned:

[<span class="timestamp">Save91%</span>, <span class="timestamp">Save84%</span>, <span class="timestamp">Save96%</span>, <span class="timestamp">Save99%</span>, <span class="timestamp">Save82%</span>]

Hope this is what you are looking for.

Answer 2

You can definitely do it with Selenium. It would be quite easy in fact. Selenium have plugins for many different programming languages, so just choose the one you know better and read Selenium documentation for that specific language.

I personally use python and it is quite easy to understand.

Here is the selenium documentation for Python .

Answer 3

I finally use a Firefox Add-on called Greasemonkey to scrape the website.

https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/

Greasemonkey can capture dynamically generated data in http://www.quickbid.com.tw/