抓取网站

Question

sHey I've just started using scrapy and was trying it out on a website "diy.com" but i cant seem to get the CrawlSpider to follow links or scrape any data. I think it might be my regex but i cant see anything

any help will be appreciated

from scrapy.spider import Spider
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from items import PartItem

class DIY_spider(CrawlSpider):
    name = 'diy_cat'
    allowed_domains = ['diy.com']

    start_urls =[
        "http://www.diy.com/nav/decor/tiles/wall-tiles"

    ]

    rules = (
        Rule(SgmlLinkExtractor(allow=(r'/(nav)/(decor)/(\w*)/(.*)(\d*)$', ),deny=(r'(.*)/(jsp)/(.*)')), callback='parse_item',follow = True),

def parse_items(self, response):
        sel = Selector(response)
        tests =[]
        test = PartItem()

        if sel.xpath('//*[@id="fullWidthContent"]/div[2]/dl/dd[1]/ul[1]/li[3]/text()') :
            price = sel.xpath('//*[@id="fullWidthContent"]/div[2]/dl/dd[1]/ul[1]/li[3]/text()')
        else:
            price= sel.xpath('//dd[@class="item_cta"]/ul[@class="fright item_price"]/li/text()').extract()
        if not price:
           return test





return test

Answer 1

Your rule states parse_item as the callback but the actual callback is named parse_items . Additionally, the indenting for the parse_items function is incorrect, but that could simply be a formatting issue when pasting the code in.

Answer 2

Besides, @Talvalin's note, you are not getting actual prices.

Try this version of parse_item :

def parse_item(self, response):
    sel = Selector(response)

    price_list = sel.xpath('//span[@class="onlyPrice"]/text()').extract()
    for price in price_list:
        if price:
            item = PartItem()
            item['price'] = price
            yield item