Scrapinghub scrapy：ModuleNotFoundError：没有名为“pandas”的模块

Question

我尝试通过命令行和 GitHub 部署到 Zyte，但我一直遇到上述错误。

我尝试了不同版本的 Scrapy 版本 1.5 到 2.5，但错误仍然存在。

我还尝试将我的 Scrapinghub.yml 设置为以下脚本，但它似乎无法解决错误

project: 557222

stacks:
    default: scrapy:1.5-py3






 Deploy log location: C:\Users\Public\Documents\Wondershare\CreatorTemp\shub_deploy_xzbjx9qc.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'
  File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy
    execute(settings=settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 249, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 336, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/usr/local/lib/python3.6/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/app/__main__.egg/greyhound_recorder_website/spiders/greyhound.py", line 7, in <module>
ModuleNotFoundError: No module named 'pandas'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}
{"status": "error", "message": "Internal error"}

Answer 1

您应该在 Scrapy Cloud 中指定需要安装的依赖项。 例如。：

scrapinghub.yml ：

project: 557222

stacks:
    default: scrapy:1.5-py3

requirements:
    file: requirements.txt

要求.txt ：

pandas

Answer 2

Pandas和Scrapy一样是python package，需要安装。 pip install pandas或访问https://pandas.pydata.org/docs/getting_started/install.html