Scrapy - How to scrape content generated with javascript?

Question

I'm trying to scrape some classified ads on http://www.head-fi.org/f/6550/headphones-for-sale-trade

I created a spider which can scrape the titles, prices, descriptions, etc. It is working well but I can't figure out how the pagination works on that specific website. I believe it is being generated with javascript? Since the URL doesn't change.

This is my code to scraping the first page

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from headfi_headphones.items import HeadfiHeadphonesItem

class MySpider(CrawlSpider):
    name = "headfiheadphones"
    allowed_domains = ["head-fi.org"]
    start_urls = ["http://www.head-fi.org/f/6550/headphones-for-sale-trade"]

    #rules = (
    #    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=("//a[@class='tooltip']",)), callback="parse_items", follow= True),
    #)

def parse(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.xpath("//tr[@class='thread']")
    items = []
    for title in titles:
        item = HeadfiHeadphonesItem()
        item["title"] = title.select("td[@class='thread-col']/div[@class='shazam']/div[@class='thumbnail_body']/a[@class='classified-title']/text()").extract()
        item["link"] = title.select("td[@class='thread-col']/div[@class='shazam']/div[@class='thumbnail_body']/a[@class='classified-title']/@href").extract()
        item["img"] = title.select("td[@class='thread-col']/div[@class='shazam']/div[@class='thumbnail']/a[@class='thumb']/img/@src").extract()
        item["saletype"] = title.select("td/strong/text()").extract()
        item["price"] = title.select("td/div[@class='price']/span[@class='ctx-price']/text()").extract()
        item["currency"] = title.select("td/div[@class='price']/span[@class='currency']/text()").extract()
        items.append(item)
    return items

It returns something like this (I've included one entry)

{"img": ["http://cdn.head-fi.org/9/92/80x80px-ZC-9228072e_image.jpeg"], "title": ["Hifiman HE1000 Mint"], "saletype": ["For Sale"], "price": ["$2,000"], "currency": ["(USD)"], "link": ["/t/819200/hifiman-he1000-mint"]},

Is there a way to scrape through each page (1-80 or so) which is being populated on a table by what I assume is javascript?

Answer 1

To properly parse Javascript you should consider using selenium . The package is available here: https://pypi.python.org/pypi/selenium .