[英]scrapy from script output in json
I am running scrapy
in a python script 我在python脚本中运行scrapy
def setup_crawler(domain):
dispatcher.connect(stop_reactor, signal=signals.spider_closed)
spider = ArgosSpider(domain=domain)
settings = get_project_settings()
crawler = Crawler(settings)
crawler.configure()
crawler.crawl(spider)
crawler.start()
reactor.run()
it runs successfully and stops but where is the result ? 它成功运行并停止但结果在哪里? I want the result in json format, how can I do that? 我希望结果采用json格式,我该怎么做?
result = responseInJSON
like we do using command 就像我们使用命令一样
scrapy crawl argos -o result.json -t json
You need to set FEED_FORMAT
and FEED_URI
settings manually: 您需要手动设置FEED_FORMAT
和FEED_URI
设置:
settings.overrides['FEED_FORMAT'] = 'json'
settings.overrides['FEED_URI'] = 'result.json'
If you want to get the results into a variable you can define a Pipeline
class that would collect items into the list. 如果要将结果输入变量,可以定义一个将项目收集到列表中的Pipeline
类。 Use the spider_closed
signal handler to see the results: 使用spider_closed
信号处理程序查看结果:
import json
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from scrapy.utils.project import get_project_settings
class MyPipeline(object):
def process_item(self, item, spider):
results.append(dict(item))
results = []
def spider_closed(spider):
print results
# set up spider
spider = TestSpider(domain='mydomain.org')
# set up settings
settings = get_project_settings()
settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1}
# set up crawler
crawler = Crawler(settings)
crawler.signals.connect(spider_closed, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
# start crawling
crawler.start()
log.start()
reactor.run()
FYI, look at how Scrapy parses command-line arguments . 仅供参考,看看Scrapy如何解析命令行参数 。
Also see: Capturing stdout within the same process in Python . 另请参阅: 在Python中的同一进程中捕获stdout 。
I managed to make it work simply by adding the FEED_FORMAT
and FEED_URI
to the CrawlerProcess
constructor, using the basic Scrapy API tutorial code as follows: 我设法通过将FEED_FORMAT
和FEED_URI
添加到CrawlerProcess
构造函数来使其工作,使用基本的Scrapy API教程代码,如下所示:
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'result.json'
})
Easy! 简单!
from scrapy import cmdline
cmdline.execute("scrapy crawl argos -o result.json -t json".split())
Put that script where you put scrapy.cfg
把那个脚本放在你放scrapy.cfg
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.