如何爬取多级页面到Scrapy中的一项？

Question

All examples I found of Scrapy talk about how to crawl a single page, or how to crawl multiple-level pages, when each of the deepest pages is saved as an independent Item . But my situation is a bit more complex.

For example, the website structure is:

A (List page of books)
--> B (Book summary page)
----> C (Book review pages)
----> D (Book download pages)

And so the definition of the Item looks like:

class BookItem(scrapy.Item):
    name = scrapy.Field()
    type = scrapy.Field()
    introduction = scrapy.Field()
    resources = scrapy.Field() # To be a list of ResourceItem
    reviews = scrapy.Field() # To be a list of ReviewItem

# Download pages
class ResourceItem(scrapy.Item):
    title = scrapy.Field()
    createDate = scrapy.Field()
    author = scrapy.Field()
    link = scrapy.Field()


# Book reviews
class ReviewItem(scrapy.Item):
    title = scrapy.Field()
    createDate = scrapy.Field()
    author = scrapy.Field()
    content = scrapy.Field()

How could I complete all fields of BookItem ? I do know that I can write 4 methods, like parse_A() , parse_B() , parse_C() and parse_D() , and Scrapy allows them to be a workflow by using yield scrapy.Request() at the end of each methods.

But what should I return in the deepest methods, ie parse_C() and parse_D() ?

1. If I return a ResourceItem or ReviewItem , it will be saved directly.
2. If I return the BookItem from upper methods, the uncompleted item will be saved directly, too.
3. If I return a Request for parse_D() in parse_C() , it will not work either, because resources may be empty (that is to say, there may be no links of C on B pages at all). So parse_C() won't be called, leaving parse_D() uncalled, and D fields unfilled in the end.

Answer 1

You can pass some data around using meta parameters (see https://docs.scrapy.org/en/latest/topics/request-response.html ).

So you can populate your item in multiple requests/parse functions.

Quick example to show the logic:

def parse_summary(self, response):
  book_item = # scrape book item here
  reviews_url = # extract reviews url 
  resources_url = # extract resources url
  return scrapy.Request(reviews_url, callback=self.parse_reviews, meta={'item': book_item, 'resources_url': resources_url })

def parse_reviews(self, response):
  book_item = response.meta.get('item') # get item draft
  book_item.reviews = # extract reviews here
  resources_url = response.meta.get('resources_url')
  return scrapy.Request(resources_url, callback=self.parse_resources, meta={'item': book_item })

def parse_resources(self, response):
  book_item = response.meta.get('item') # get item draft
  book_item.ressources = # extract ressources here
  return book_item # once completed, return the item

Hope you get the idea (I'm not really confident on the code execution, just wrote it down without testing).

Answer 2

I can answer it by myself now.

Just yield None or omit return statements in parse_C() and parse_D() will solve the problem.

Some explanation

Scrapy will not close the spider simply because one of the callbacks returns nothing, but ensure that there is no new one in the request queue as well.

So, since parse_B() will not return None or Item before it completes yielding all of its requests of Subpage C & D, the workflow won't be interrupted.