您如何让scrapy产生所有物品？

Question

I just learned scrapy recently. How do you let scrapy yield all items?

For example, if I want to extract a book . And the home page is the book title , layer two is the chapter , layer three is the article .

class BookSpider(scrapy.spider.Spider):
   name = 'book'
   allowed_domains = ['book.com']
   start_urls = ['http://www.book.com']

   def __init__(self):
      self.items = []

   def parse(self, response):
      link = response.xpath('//chapter').extract()

      for l in links:
         yield Request(l, callback=self.parse_chapter)

      print self.items

   def parse_chapter(self, response):
      link = response.xpath('//article').extract()

      for l in links:
         yield Request(l, callback=self.parse_article)
      return

   def parse_article(self, response):
      item = BookItem()
      item['article'] = response.url
      self.items.append(item)
      return

But the result is just an empty list. Why is self.items not able to be built?

Answer 1

You need to return or yield item or a list of items from any of the callbacks:

def parse_article(self, response):
    item = BookItem()
    item['article'] = response.url
    return item