Spider用于在Scrapy中解析JSON

Question

I have been using scrapy for some time now and I wish to know what is the best way to parse JSON. There are specially designed spiders for XML and CSV in scrapy viz. XMLFeedSpider and CSVFeedSpider . Is there any such spider for JSON too ?

Answer 1

There is no default JSON parser for Scrapy, but you can use the json module to parse your response and convert it to items:

jsonresponse = json.loads(response.body_as_unicode())

However if you plan to write a general solution like CSVFeedSpider feel free to contribute it to the Scrapy project but I guess json.loads does the trick of consuming JSON responses.