I am using Scrapy with several Spiders, and need custom json output, that would include some Spider stats (list of successful requests, list of errors, etc). I have made custom item pipeline, but I don't know how to access stats from there. This is my pipeline code so far:
class JsonWithEncodingPipeline(object):
def open_spider(self, spider):
self.file = codecs.open(spider.output_path, 'w', encoding='utf-8')
def process_item(self, item, spider):
line = json.dumps(dict(item), ensure_ascii=False, indent=2) + "\n"
self.file.write(line)
return item
def spider_closed(self, spider):
self.file.close()
You can access stats like this:
class MyPipeline:
def __init__(self, stats):
self.stats = stats
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.stats)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.