Running Scrapy spider from a script with scrapy arguments

Question

I have written a spider in scrapy & I'm successfully able to run it via a python script. The problem is, while running a spider via scrapy prompt we supply agruments like " -o ScrapedData.json -t json " I want to run "scrapy crawl myspider -o ScrapedData.json -t json " via python script. Any help is highly appreciated. Thanks in advance!

Answer 1

• -o ScrapedData.json overrides the FEED_URI setting
• and -t json overrides the FEED_FORMAT setting

(See https://github.com/scrapy/scrapy/blob/master/scrapy/commands/crawl.py#L24 )

So, when running your spider from a script, you should look at overriding these 2 settings.

Answer 2

You mean like this?

from scrapy import cmdline

cmdline.execute("scrapy crawl myspider -o ScrapedData.json -t json".split())