如何从同一个 python 脚本运行 2 个爬虫

Question

我有两个可以独立运行的 python 爬虫。

crawler1.py crawler2.py

它们是我想要运行的分析的一部分，我想将所有内容导入到 commong 脚本中。

from crawler1.py import * 
from crawler2.py import * 

在我的脚本中有点低我有这样的东西

if <condition1>:
    // running crawler1 
    runCrawler('crawlerName', '/dir1/dir2/')

if <condition2>:
    // running crawler2 
    runCrawler('crawlerName', '/dir1/dir2/')

runCrawler 是：

def runCrawler(crawlerName, crawlerFileName):
    print('Running crawler for ' + crawlerName)

    process = CP(
        settings={
            'FEED_URI'   : crawlerFileName,
            'FEED_FORMAT': 'csv'
        }
    )

    process.crawl(globals()[crawlerName])
    process.start()

我收到以下错误：

Exception has occurred: ReactorAlreadyInstalledError
reactor already installed

第一个爬虫运行正常。 第二个有问题。 有任何想法吗？

我通过视觉工作室调试器运行上述内容。

Answer 1

你的代码应该是

from twisted.internet import reactor
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging


# your code




settings={
            'FEED_FORMAT': 'csv'
        }
process = CrawlerRunner(Settings)

if condition1:
    process.crawl(spider1,crawlerFileName=crawlerFileName) 
if condition2:
    process.crawl(spider2,crawlerFileName=crawlerFileName)

d = process.join()
d.addBoth(lambda _: reactor.stop())
reactor.run()  # it will run both crawlers and code inside the function

你的蜘蛛应该像

class spider1(scrapy.Spider):
    name = "spider1"
    custom_settings = {'FEED_URI'   : spider1.crawlerFileName}
    def start_requests(self):
            yield scrapy.Request('https://scrapy.org/') 

    def parse(self, response):
        pass