Scrapy-从python脚本运行蜘蛛

Question

I am trying to run scrapy from a python script according to documentation http://scrapy.readthedocs.io/en/0.16/topics/practices.html

def CrawlTest():

    spider = PitchforkSpider(domain='"pitchfork.com"')
    crawler = Crawler(Settings())
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()
    log.start()
    reactor.run() # the script will block here

but when I run it, I get the following error:

AttributeError: 'Settings' object has no attribute 'update_settings'

has something been deprecated? what is wrong here?

my version is Scrapy 1.1.2

Answer 1

You are looking at Scrapy 0.16 docs but using Scrapy 1.1.2.

Here is the correct documentation page .

FYI, you should now be using CrawlerProcess or CrawlerRunner .