Make scrapy export to csv

Question

I want to use scrapy in the following way

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' })

process.crawl(my_super_scraper) 
process.start()

It works with my_super_scraper , but I cannot figure out how to export to CSV. I cannot find it in the documentation either.

Answer 1

You need to set FEED_FORMAT and FEED_URI parameters as follow:

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)', 
    'FEED_FORMAT': 'CSV', 
    'FEED_URI': 'file:///tmp/export.csv',
})

process.crawl(my_super_craper)
process.start()

More information about feed export here https://docs.scrapy.org/en/latest/topics/feed-exports.html

This post shows how to export to JSON format: Scrapy process.crawl() to export data to json