ImportError：沒有名為管道的模塊-Scrapy / PyInstaller

Question

我正在嘗試從Scrapy項目創建一個exe文件。 我不得不添加數十個隱藏的導入以避免錯誤，但是現在我得到了ImportError: No module named pipelines而且我不知道該怎么做。

bot/
    engine_bot/
        engine_bot/
            spiders/
                __init__.py
                main_spider.py
            __init__.py
            items.py
            middlewares.py
            pipelines.py
            settings.py
            utils.py
        __init__.py
        helper.py
    main.py
    __init__.py

Main.py：

from scrapy.crawler import CrawlerProcess
from engine_bot.engine_bot.spiders.main_spider import MainSpider


if __name__ == '__main__':
    process = CrawlerProcess({'BOT_NAME':'engine_bot',
    'SPIDER_MODULES':['engine_bot.engine_bot.spiders'],
    'NEWSPIDER_MODULE':'engine_bot.engine_bot.spiders',
    'ROBOTSTXT_OBEY':False,
    'DOWNLOAD_DELAY':0.20,
    'LOG_FILE':'scrapy.log',
    'LOG_LEVEL':'DEBUG',
    'ITEM_PIPELINES':{
    'engine_bot.engine_bot.pipelines.XmlExportPipeline': 300,

    }
    })
    process.crawl(MainSpider)
    process.start()

celery.log：

2017-11-03 14:01:47 [twisted] CRITICAL: Unhandled error in Deferred:
2017-11-03 14:01:47 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
  File "site-packages\scrapy\crawler.py", line 95, in crawl
  File "site-packages\scrapy\crawler.py", line 77, in crawl
  File "site-packages\scrapy\crawler.py", line 102, in _create_engine
  File "site-packages\scrapy\core\engine.py", line 70, in __init__
  File "site-packages\scrapy\core\scraper.py", line 71, in __init__
  File "site-packages\scrapy\middleware.py", line 58, in from_crawler
  File "site-packages\scrapy\middleware.py", line 34, in from_settings
  File "site-packages\scrapy\utils\misc.py", line 44, in load_object
  File "importlib\__init__.py", line 37, in import_module
ImportError: No module named pipelines

PyInstaller CMD（很多隱藏的導入，試圖添加scrapy.pipelines）

 pyinstaller main.py --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions.closespider 

--hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions.closespider --hidden-進口scrapy.extensions.feedexport-隱藏進口scrapy.extensions.logstats-隱藏進口scrapy.spiderloader-隱藏進口scrapy.statscollectors-隱藏進口scrapy.logformatter-隱藏進口scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions.closespider --hidden-import scrapy .extensions.feedexport-隱藏的進口scrapy.extensions.telnet-隱藏的進口scrapy.spiderloader-隱藏的進口scrapy.statscollectors-隱藏的進口scrapy.logformatter-隱藏的進口scrapy.extensions.closespider- hidden-import scrapy.extensions.feedexport-隱藏進口scrapy.extensions.memusage --hidden-impo rt scrapy.spiderloader-隱藏進口scrapy.statscollectors-隱藏進口scrapy.logformatter-隱藏進口scrapy.extensions.closespider-隱藏進口scrapy.extensions.feedexport-隱藏進口scrapy.extensions.logstats --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy .spiderloader-隱藏進口scrapy.stats收集器-隱藏進口scrapy.logformatter-隱藏進口scrapy.extensions.closespider-隱藏進口scrapy.extensions.corestats-隱藏進口scrapy.extensions.spiderstate- hidden-import scrapy.extensions.throttle-隱藏導入scrapy.core.scheduler --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth-隱藏導入scrapy.downloadermiddlewares.downloadtimeout-隱藏導入scrapy.downlo adermiddlewares.useragent-隱藏進口scrapy.downloadermiddlewares.defaultheaders-隱藏進口scrapy.downloadermiddlewares.ajaxcrawl-隱藏進口scrapy.downloadermiddlewares.retry-隱藏進口scrapy.downloadermiddlewares.redirect-隱藏進口scrapy。 downloadermiddlewares.cookies-隱藏導入scrapy.downloadermiddlewares.http壓縮--hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy。 downloadermiddlewares.chunked-隱藏導入scrapy.downloadermiddlewares.com解壓縮--hidden-import scrapy.downloadermiddlewares.httperror --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy。 spidermiddlewares.depth-隱藏導入scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-impor t scrapy.spidermiddlewares.urllength-隱藏導入scrapy.pipelines-隱藏導入engine_bot.pipelines

我真的不知道該怎么辦，有什么建議嗎？

Answer 1

我認為問題在於Main.py這一部分：

'ITEM_PIPELINES':{
    'engine_bot.engine_bot.pipelines.XmlExportPipeline': 300,
}

根據提供的項目結構，應閱讀

'ITEM_PIPELINES':{
    'engine_bot.pipelines.XmlExportPipeline': 300,
}

即在路徑上少一個engine_bot 。

Answer 2

從更改項目結構

bot/
    engine_bot/
        engine_bot/
            spiders/
                __init__.py
                main_spider.py
            __init__.py
            items.py
            middlewares.py
            pipelines.py
            settings.py
            utils.py
        __init__.py
        helper.py
    main.py
    __init__.py

至

bot/
    some_new_name/
        engine_bot/
            spiders/
                __init__.py
                main_spider.py
            __init__.py
            items.py
            middlewares.py
            pipelines.py
            settings.py
            utils.py
        __init__.py
        helper.py
    main.py
    __init__.py

請參閱我將engine_bot重engine_bot為some_new_name