处理 Scrapy 中的所有产量项

Question

Currently I have a Scrapy Spider yielding various items on the parse method. Is there any way to get all items that have been yield, regardles of how many times the parse method has been called?

Answer 1

Using pipeline you'll be able to accumulate all items in an array like structure ( process_item in your pipeline):

self.items.append(item) # I use self.items class variable defined at your pipeline

and process all of them in spider_closed .

Answer 2

I am unsure about what you mean by get the items. If you want to export them into a file you can use the feed export , by executing the spider like:

scrapy crawl my_spider -o my_data.csv

It supports other extensions, check the link for those.

From your title it seems you want to process the yielded items, in that case you need an ItemPipeline . From the docs:

After an item has been scraped by a spider, it is sent to the Item Pipeline which processes it through several components that are executed sequentially.

...

Typical uses of item pipelines are:

• cleansing HTML data

• validating scraped data (checking that the items contain certain fields)

• checking for duplicates (and dropping them)

• storing the scraped item in a database

You can also see some pipelines examples here .

Both methods operate independently of how many times parse method has been called.

Answer 3

There are generally two ways to do this.

Firstly, you can simply save the output in the JSON file using command scrapy crawl my_spider -o my_data.json . Secondly, you can write a pipeline and store the output in any DB to structure you want.