為什么我在Scrapy中遇到KeyError？

Question

我在芹菜中使用Scrapy蜘蛛，並且隨機收到此類錯誤

Unhandled Error
Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 428, in fireEvent
      DeferredList(beforeResults).addCallback(self._continueFiring)
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 321, in addCallback
      callbackKeywords=kw)
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 310, in addCallbacks
      self._runCallbacks()
    File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
      current.result = callback(current.result, *args, **kw)
  --- <exception caught here> ---
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 441, in _continueFiring
      callable(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 667, in disconnectAll
      selectables = self.removeAll()
    File "/usr/lib/python2.7/site-packages/twisted/internet/epollreactor.py", line 191, in removeAll
      [self._selectables[fd] for fd in self._reads],
  exceptions.KeyError: 94

該數字因大小寫而異（在另一種情況下，94可能是97，依此類推）

我在用：

celery==3.1.19
Django==1.9.4
Scrapy==1.3.0

這是我在Celery中運行Scrapy的方式：

from billiard import Process
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

class MyCrawlerScript(Process):
    def __init__(self, **kwargs):
        Process.__init__(self)
        settings = get_project_settings('my_scraper')
        self.crawler = CrawlerProcess(settings)
        self.spider_name = kwargs.get('spider_name')
        self.kwargs = kwargs

    def run(self):
        self.crawler.crawl(self.spider_name, qwargs=self.kwargs)
        self.crawler.start()

def my_crawl_manager(**kwargs):
    crawler = MyCrawlerScript(**kwargs)
    crawler.start()
    crawler.join()

在芹菜任務中，我正在打電話：

my_crawl_manager(spider_name='my_spider', url='www.google.com/any-url-here')

請知道為什么會這樣嗎？

Answer 1

我曾經有這個問題。

檢查您是否有一個空文件__init__.py文件spiders文件夾或。 它應該在那里。