Scrapy CSS选择器

Question

I am learning how to use scrapy but I am having some issue. I wrote this code, following an online tutorial, to understand a bit more about it.

import scrapy

class BrickSetSpider(scrapy.Spider):
name = 'brick_spider'
start_urls = ['http://brickset.com/sets/year-2016']

def parse(self, response):
    SET_SELECTOR = '.set'
    for brickset in response.css(SET_SELECTOR):

        NAME_SELECTOR = 'h1 a ::text'
        PIECES_SELECTOR = './/dl[dt/text() = "Pieces"]/dd/a/text()'
        MINIFIGS_SELECTOR = './/dl[dt/text() = "Minifigs"]/dd[2]/a/text()'
        PRICE_SELECTOR  =  './/dl[dt/text() = "RRP"]/dd[3]/text()'
        IMAGE_SELECTOR = 'img ::attr(src)'
        yield {
            'name': brickset.css(NAME_SELECTOR).extract_first(),
            'pieces': brickset.xpath(PIECES_SELECTOR).extract_first(),
            'minifigs': brickset.xpath(MINIFIGS_SELECTOR).extract_first(),
    'retail price': brickset.xpath(PRICE_SELECTOR).extract_first(),
            'image': brickset.css(IMAGE_SELECTOR).extract_first(),
        }

    NEXT_PAGE_SELECTOR = '.next a ::attr(href)'
    next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
    if next_page:
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )

Since the sites divide the product listed in years and this code crawls just data from 2016 I decided to extend it and analyze also the data of previous years. The idea of the code is this:

PREVIOUS_YEAR_SELECTOR = '...'
previous_year= response.css(PREVIOUS_YEAR_SELECTOR).extract_first()
if previous_year:
    yield scrapy.Request(
        response.urljoin(previous_year),
                callback=self.parse
            )

I tried different things but I really have no idea of what to write instead of '...' I also tried with xpath but nothing seems to work.

Answer 1

Maybe you want to exploit the structure of the href attribute? It seems to follow the structure /sets/year-YYYY . By this you can use a regex based selector or - if you are lazy like my - just a contains() :

XPath: //a[contains(@href,"/sets/year-")]/@href

I'm not sure if this is also possible with CSS. So the ... can be filled with:

PREVIOUS_YEAR_SELECTOR_XPATH = '//a[contains(@href,"/sets/year-")]/@href'
previous_year = response.xpath(PREVIOUS_YEAR_SELECTOR).extract_first()

But I think you will go for ALL years, so maybe you want to loop over the links:

PREVIOUS_YEAR_SELECTOR_XPATH = '//a[contains(@href,"/sets/year-")]/@href'
for previous_year in response.xpath(PREVIOUS_YEAR_SELECTOR):
    yield scrapy.Request(response.urljoin(previous_year), callback=self.parse)

I think you are on a good way. Google for an CSS/XPATH cheat sheet that matches your needs and checkout the FirePath extension or similar. It speeds up the selector setup a lot :)

Answer 2

You have at least two options here. The first is to use generic CrawlSpider and define which links you want to extract and follow. Something like this:

class BrickSetSpider(scrapy.CrawlSpider):
    name = 'brick_spider'
    start_urls = ['http://brickset.com/sets']
    rules = (
        Rule(LinkExtractor(
            allow=r'\/year\-[\d]{4}'), callback='parse_bricks', follow=True),
    )
#Your method renamed to parse_bricks goes here

Note: you need to rename parse method to some other name like 'parse_bricks' since the CrawlSpider uses the parse method itself.

The second options are to set start_urls to a page http://brickset.com/browse/sets containing all links to year sets and add a method to parse those links:

class BrickSetSpider(scrapy.Spider):
    name = 'brick_spider'
    start_urls = ['http://brickset.com/browse/sets']

    def parse(self, response):
        links = response.xpath(
            '//a[contains(@href, "/sets/year")]/@href').extract()
        for link in links:
            yield scrapy.Request(response.urljoin(link), callback=self.parse_bricks)

    # Your method renamed to parse_bricks goes here