项目管道无法正常运行

Question

I have written the following code and I found that item pipeline will not work if I write in the following way, the process_item (in item pipeline) will not be executed.

class Spider(scrapy.Spider):
    name = “***”
    def __init__(self, url='http://example.com/', **kw):
        super(Spider,self).__init__(**kw)
        self.url = url 
        self.allowed_domains = [re.sub(r'^www\.', '', urlparse(url).hostname)]

    def start_requests(self):
        #return [Request(self.url, callback=self.parse, dont_filter=False)]
        return [Request(self.url, callback=self.find_all_url, dont_filter=False)]

    def find_all_url(self,response):
        log.msg('current url: '+response.url, level=log.DEBUG)
        if True:
              self.parse(response)

    def parse(self, response):
        dept = deptItem()
        dept['deptName'] = response.xpath('//title/text()').extract()[0].strip()
        dept['url'] = response.url
        log.msg('find an item: '+ str(response.url) +'\n going to return item' , level = log.INFO)
        return dept        

However, if I change the callback in start_requests from self.find_all_url to self.parse (see above the commented code), the item pipeline works, I try to find out why, but I couldn't, anyone can help?

Answer 1

I have found out that if I want to write in this way, I need to add return in front of self.parse(response) in function find_all_url .

But I am not very clear why this is the case, I guess the returned item should eventually return to the initial requests?

Answer 2

Can you post your settings?

You must define the pipelines in settings.py

ITEM_PIPELINES = {
   'MySpider.pipelines.SomePipeline': 300,
}

Basic example: https://github.com/scrapy/dirbot