[英]ModuleNotFoundError: No module named 'itemloaders' whilst deploying to scraping hub?
I'm trying to deploy my spider to scraping hub but I keep running into this error with one of my imports.我正在尝试将我的蜘蛛部署到抓取集线器,但我的一个导入一直遇到这个错误。 This works perfectly fine in development, just not when I attempt to deploy to scraping hub.
这在开发中工作得非常好,只是在我尝试部署到抓取中心时不行。
Import from item_loaders.py:从 item_loaders.py 导入:
from itemloaders.processors import Identity, MapCompose, TakeFirst, Join
from w3lib.html import remove_tags
from scrapy.loader import ItemLoader
-- --
Deploy log location: C:\Users\denzel\AppData\Local\Temp\shub_deploy_nzoj_krm.log
Output dir not removed: C:\Users\denzel\AppData\Local\Temp\shub-deploy-wsvzk2w3
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'
cmd.crawler_process = CrawlerProcess(settings)
File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 267, in __init__
super(CrawlerProcess, self).__init__(settings)
File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 145, in __init__
self.spider_loader = _get_spider_loader(settings)
File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 347, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 61, in from_settings
return cls(settings)
File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 25, in __init__
self._load_all_spiders()
File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
for module in walk_modules(name):
File "/app/python/lib/python3.8/site-packages/scrapy/utils/misc.py", line 73, in walk_modules
submod = import_module(fullpath)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
File "<frozen zipimport>", line 259, in load_module
File "/app/__main__.egg/remind_me_scraper/spiders/listing_spider.py", line 10, in <module>
File "<frozen zipimport>", line 259, in load_module
File "/app/__main__.egg/remind_me_scraper/item_loaders.py", line 2, in <module>
ModuleNotFoundError: No module named 'itemloaders'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}
{"status": "error", "message": "Internal error"}
-- --
└───remind_me_scraper
│ items.py
│ item_loaders.py
│ middlewares.py
│ pipelines.py
│ requirements.txt
│ scrapyd.conf
│ settings.py
│ test.txt
│ __init__.py
│
├───spiders
│ │ listing_spider.py
│ │ __init__.py
Welp, I figured it out, I forgot to add the itemloaders module to my requirements.txt like such: Welp,我想通了,我忘了将 itemloaders 模块添加到我的 requirements.txt 中,如下所示:
itemloaders==1.0.4
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.