沙皮跑2蜘蛛

Question

如何連續運行2個Spider？ 運行此命令將運行第一個蜘蛛，但不運行第二個。 有沒有辦法等待一個人完成？：

from scrapy import cmdline

cmdline.execute("scrapy crawl spider1".split())

cmdline.execute("scrapy crawl spider2".split())

Edit1：我使用.wait（）將其更改為：

spider1 = subprocess.Popen(cmdline.execute("scrapy crawl spider1".split()))
spider1.wait()

spider2 = subprocess.Popen(cmdline.execute("scrapy crawl spider2".split()))
spider2.wait()

我做錯了嗎，因為它只會運行第一個

編輯2：

Traceback (most recent call last):
  File "/usr/bin/scrapy", line 9, in <module>
    load_entry_point('Scrapy==0.24.6', 'console_scripts', 'scrapy')()
  File "/usr/lib/pymodules/python2.7/scrapy/cmdline.py", line 109, in execute
    settings = get_project_settings()
  File "/usr/lib/pymodules/python2.7/scrapy/utils/project.py", line 60, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "/usr/lib/pymodules/python2.7/scrapy/settings/__init__.py", line 109, in setmodule
    module = import_module(module)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named settings
1

Answer 1

我將使用具有.wait（）函數的Subprocess。 或者，您可以在子scrapy crawl使用.call() ，該子scrapy crawl會自動等待並打印出來，以從調用scrapy crawl來獲取終端文本。

spider1 = subprocess.call(["scrapy", "crawl", "spider1"])
print spider1

spider2 = subprocess.call(["scrapy", "crawl", "spider2"])
print spider2

此方法將自動等待，直到第一個蜘蛛完成，然后調用秒蜘蛛