Raise close spider from Scrapy pipeline

Question

I need to raise CloseSpider from a Scrapy Pipeline. Either that or return some parameter from the Pipeline back to the Spider to do the raise.

For example, if the date already exists raise CloseSpider:

raise CloseSpider('Already been scraped:' + response.url)

Is there a way to do this?

Answer 1

As from scrapy docs, CloseSpider Exception can only be raised from a callback function (by default parse function) in a Spider only. Raising it in pipeline will crash spider. To achieve the similar results from a pipeline, you can initiate a shutdown signal, that will close scrapy gracefully.

from scrapy.project import crawler  
crawler._signal_shutdown(9,0)

Do remember ,scrapy might process already fired or even scheduled requests even after initiating shutdown signal.

To do it from Spider, set some variable in Spider from Pipeline like this.

def process_item(self, item, spider):
    if some_condition_is_met:
        spider.close_manually = True

After this in the callback function of your spider , you can raise close spider exception.

def parse(self, response):
    if self.close_manually:
        raise CloseSpider('Already been scraped.')

Answer 2

I prefer the following solution.

class MongoDBPipeline(object):

def process_item(self, item, spider):
    spider.crawler.engine.close_spider(self, reason='duplicate')

Source: Force spider to stop in scrapy