如何使用scrapy和pyinstaller在Windows 10中创建单个可执行文件？

Question

我创建了一个scrapy蜘蛛并使用带有光盘文件夹的pyinstaller成功转换为Windows可执行文件。

为了做到这一点，我必须在scrapy site-packages中做一些细微的改变，并将这些包添加到windows光盘文件夹中，它工作得很好，

如何将其与光盘文件夹中注释的scrapy包一起制作成单个exe？

我已经在 pyinstaller 中尝试过 --OneFile 命令，但它显示了scrapy错误？

Answer 1

这里讨论的问题非常相似： python scrapy conversion to exe file using pyinstaller

最初我使用auto-py-to-exe包（实际上是pyinstaller GUI）
我在auto-py-to-exe -> advanced settings -> hidden import添加了以下行：

scrapy.spiderloader,scrapy.statscollectors,scrapy.logformatter,scrapy.extensions,scrapy.extensions.corestats,scrapy.extensions.corestats,scrapy.extensions.telnet,scrapy.extensions.memusage,scrapy.extensions.memdebug,scrapy.extensions.closespider,scrapy.extensions.feedexport,scrapy.extensions.logstats,scrapy.extensions.spiderstate,scrapy.extensions.throttle,scrapy.core.scheduler,scrapy.squeues,queuelib,scrapy.core.downloader,scrapy.downloadermiddlewares,scrapy.downloadermiddlewares.robotstxt,scrapy.downloadermiddlewares.httpauth,scrapy.downloadermiddlewares.downloadtimeout,scrapy.downloadermiddlewares.defaultheaders,scrapy.downloadermiddlewares.useragent,scrapy.downloadermiddlewares.retry,scrapy.downloadermiddlewares.ajaxcrawl,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.httpcompression,scrapy.downloadermiddlewares.redirect,scrapy.downloadermiddlewares.cookies,scrapy.downloadermiddlewares.httpproxy,scrapy.downloadermiddlewares.stats,scrapy.downloadermiddlewares.httpcache,scrapy.spidermiddlewares,scrapy.spidermiddlewares.httperror,scrapy.spidermiddlewares.offsite,scrapy.spidermiddlewares.referer,scrapy.spidermiddlewares.urllength,scrapy.spidermiddlewares.depth,scrapy.pipelines,scrapy.dupefilters,scrapy.core.downloader.handlers.datauri,scrapy.core.downloader.handlers.file,scrapy.core.downloader.handlers.http,scrapy.core.downloader.handlers.s3,scrapy.core.downloader.handlers.ftp,scrapy.core.downloader.webclient,scrapy.core.downloader.contextfactory

在最后一个文本框中出现以下命令之后（不要忘记更改脚本的路径）：

pyinstaller -y -F --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/script.py"

如果在此之后您的命令返回： ImportError: No module named 'modulename' - 将丢失的模块添加到隐藏导入并使用新的扩展隐藏导入重复此过程。
（我重复了这个过程 48 次以接收工作 exe 文件（并接收 sumbodules 列表）！！）

Answer 2

通过在规范文件中使用--hidden 导入来修复它。 Pyinstaller 不支持scrapy 中的所有二级模块导入。

运行 pyinstaller 命令，只需使用以下隐藏的导入更改更新规范文件，

hiddenimports=['scrapy.spiderloader','scrapy.statscollectors','scrapy.logformatter','scrapy.extensions','scrapy.extensions.logstats', 'scrapy.extensions.corestats','scrapy.extensions.memusage','scrapy.extensions.feedexport','scrapy.extensions.memdebug', 'scrapy.extensions.closespider','scrapy.extensions.throttle','scrapy.extensions.telnet','scrapy.extensions.spiderstate', 'scrapy.core.scheduler','scrapy.core.downloader','scrapy.downloadermiddlewares','scrapy.downloadermiddlewares.robotstxt', 'scrapy.downloadermiddlewares.httpauth','scrapy.downloadermiddlewares.downloadtimeout','scrapy.downloadermiddlewares.defaultheaders', 'scrapy.downloadermiddlewares.useragent','scrapy.downloadermiddlewares.retry','scrapy.core.downloader.handlers.http', 'scrapy.core.downloader.handlers.s3','scrapy.core.downloader.handlers.ftp','scrapy.core.downloader.handlers.datauri', 'scrapy.core.downloader.handlers.file','scrapy.downloadermiddlewares.ajaxcrawl','scrapy.core.downloader.contextfactory', 'scrapy.downloadermiddlewares.redirect','scrapy.downloadermiddlewares.httpcompression','scrapy.downloadermiddlewares.cookies', 'scrapy.downloadermiddlewares.httpproxy','scrapy.downloadermiddlewares.stats','scrapy.downloadermiddlewares.httpcache', 'scrapy.spidermiddlewares','scrapy.spidermiddlewares.httperror','scrapy.spidermiddlewares.offsite','scrapy.spidermiddlewares.referer', 'scrapy.spidermiddlewares.urllength','scrapy.spidermiddlewares.depth','scrapy.pipelines','scrapy.dupefilters','queuelib', 'scrapy.squeues',]

修复了 45 个模块导入问题。 使用--onefile有助于在单个可执行文件中运行 scrapy 项目。 希望任何人都觉得它有用。

Answer 3

按照更新的文档使您的爬虫蜘蛛成为 python 脚本！

按照通常的 pyinstaller 命令生成可执行文件（确保您是从 scrapy 项目中运行它）。

    pyinstaller --onefile filename.py