Scrapy 不抓取页面上的所有项目

Question

I'm scraping a e-commerce site which has 48 products on each page except the last page.

I'm using Scrapy for this. The problem is, it is not scraping all products from the page. For example, it scrapes 12 from page 1, 18 from 2, 10 from 3, 19 from 4, and so on. It should scrape all 48 products from each page, but it doesn't.

Below is my script. For the last 2 days, I can't figure out what am I doing wrong.

UPDATE I deduped the url list before scraping and added log messages to find out what the issue is. Current code:

import scrapy
from productspider.items import Product
from urlparse import urlparse


class Ecommerce(scrapy.Spider):
    name = "ecommerce"

    def __init__(self, *args, **kwargs):
        urls = kwargs.pop('urls', [])
        if urls:
            self.start_urls = urls.split(',')
        self.logger.info(self.start_urls)
        super(Ecommerce, self).__init__(*args, **kwargs)

    page = 1
    parse_product_called = 0

    def parse(self, response):

        url = response.url
        if url.endswith('/'):
            url = url.rstrip('/')

        o = urlparse(url)

        products = response.xpath(
            "//a[contains(@href, '" + o.path + "/products/')]/@href").extract()

        if not products:
            raise scrapy.exceptions.CloseSpider("All products scraped")

        products = dedupe(products)

        self.logger.info("Products found on page %s = %s" % (self.page, len(products)))
        for product in products:
            yield scrapy.Request(response.urljoin(product), self.parse_product)

        self.page += 1
        next_page = o.path + "?page=" + str(self.page)
        yield scrapy.Request(response.urljoin(next_page), self.parse)

    def parse_product(self, response):

        self.parse_product_called += 1
        self.logger.info("Parse product called %s time" % self.parse_product_called)

        product = Product()
        product["name"] = response.xpath(
            "//meta[@property='og:title']/@content")[0].extract()
        product["price"] = response.xpath(
            "//meta[@property='og:price:amount']/@content")[0].extract()

        return product

def dedupe(seq, idfun=None):
   if idfun is None:
       def idfun(x): return x
   seen = {}
   result = []
   for item in seq:
       marker = idfun(item)
       if marker in seen: continue
       seen[marker] = 1
       result.append(item)
   return result

Scrapy log after crawling:

2017-12-30 13:18:55 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 86621,
 'downloader/request_count': 203,
 'downloader/request_method_count/GET': 203,
 'downloader/response_bytes': 10925361,
 'downloader/response_count': 203,
 'downloader/response_status_count/200': 203,
 'finish_reason': 'All products scraped',
 'finish_time': datetime.datetime(2017, 12, 30, 7, 48, 55, 370000),
 'item_scraped_count': 193,
 'log_count/DEBUG': 397,
 'log_count/INFO': 210,
 'request_depth_max': 9,
 'response_received_count': 203,
 'scheduler/dequeued': 203,
 'scheduler/dequeued/memory': 203,
 'scheduler/enqueued': 418,
 'scheduler/enqueued/memory': 418,
 'start_time': datetime.datetime(2017, 12, 30, 7, 48, 22, 405000)}
2017-12-30 13:18:55 [scrapy.core.engine] INFO: Spider closed (All products scraped)

And the log messages:

2017-12-30 13:18:25 [ecommerce] INFO: Products found on page 1 = 48

2017-12-30 13:18:32 [ecommerce] INFO: Products found on page 2 = 48

2017-12-30 13:18:35 [ecommerce] INFO: Products found on page 3 = 48

2017-12-30 13:18:38 [ecommerce] INFO: Products found on page 4 = 48

2017-12-30 13:18:41 [ecommerce] INFO: Products found on page 5 = 48

2017-12-30 13:18:43 [ecommerce] INFO: Products found on page 6 = 48

2017-12-30 13:18:45 [ecommerce] INFO: Products found on page 7 = 48

2017-12-30 13:18:48 [ecommerce] INFO: Products found on page 8 = 48

2017-12-30 13:18:51 [ecommerce] INFO: Products found on page 9 = 24

The log "Parse product called" was printed each time parse_product was called. The last log message is:

2017-12-30 13:18:55 [ecommerce] INFO: Parse product called 193 time

As you can see, it found a total of 408 products but called the parse_product function just 193. Hence only 193 items were scraped.

Answer 1

Two issues in your code

Shutting the scraper down

if not products:
   raise scrapy.exceptions.CloseSpider("All products scraped")

Using above you request the spider to terminate as soon as it can. This is not a good thing to do. This is only used when you don't want the scraping to continue

Not ending the scraper

self.page += 1
next_page = o.path + "?page=" + str(self.page)
yield scrapy.Request(response.urljoin(next_page), self.parse)

You have an uncontrolled paging logic which needs to end. So you can use the fact that any page which doesn't have 48 products is the last page

self.page += 1
next_page = o.path + "?page=" + str(self.page)
if len(products) == 48:
   yield scrapy.Request(response.urljoin(next_page), self.parse)