如何在scrapy中存储json文件？

Question

I am currently using Scrapy to crawl some domains from different website and I wonder how to save my data in a local json file with the format of either a list or a dictionary with the key of 'domain' and a list of domains as value.

In the crawler file, the item is like this:

item['domain'] = 'xxx'.extract()
yield item

import json
import codecs

class ChinazPipeline(object):

    def __init__(self):
        self.file = codecs.open('save.json', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        line = json.dumps(dict(item), ensure_ascii=False) + "\n"
        self.file.write(line)
        return item

What I expect is:

{"domain": "['google.com', 'cnn.com', 'yahoo.com']"}

or just simply save all domains that I crawled as a list in json, either way works for me.

Answer 1

It's rather simple. Json is default scrapy exporter. You can use it by turning on output to JSON file:

scrapy runspider yourspider.py -o filename.json 

Scrapy will automatically determine format you with to have by file type. Other options are .csv and .jsonline .

It's an easy way. Otherwize you can write your own ItemExporter . Take a look at exporters documentation .

NB:

You don't even need to open file during spider initiation, scrapy will manage it by itself. Just yield items and scrapy will write it to file automatically.

---

Scrapy is most suitable for one page -> one item schema. What you want is scrape all items in advance and then export them as single list.

So you should have some variable like self.results , append there new domains from every process_item() call. And then export it on spider close event. There's shortcut for this signal. So you can just add:

def closed(self, reason):
    # write self.results list to JSON file.

More documentation on Spider.closed() method .