堆栈内存溢出
  繁体   English   中英

在scrapy的项目中导入自定义包的麻烦

[英]Troubles importing customed packages in scrapy's project

 原文  2022-07-01 16:58:07  6   0    python/ scrapy/ pycharm/ virtualenv/ python-import

问题描述

我已经阅读了有关相对导入的文档，但它对我的谦虚技能没有帮助。

我的树如下：

GUI (dir)
 ├── __init__.py (file)
 ├── arachnides (dir) # a scrapy project
 │   ├── __init__.py (file)
 │   ├── scrapy.cfg (file)
 │   ├── .scrapy (dir)
 │   └── arachnides (dir)
 │        ├── __init__.py (file)
 │        ├── items.py (file)
 │        └── spiders (dir)
 │              ├── __init__.py (file)
 │              └── araignees.py (file)
 │ 
 └── datavacuum (dir)
     ├── __init__.py (file)
     ├── database.py (file)
     └── parse.py (file)

请注意，我使用 Pycharm 作为 IDE，并且按照scrapy's doc中的建议，使用 Python 3.10 在名为Turf的特定 Conda 环境中工作。
 出于某些目的，我更喜欢datavacuum和arachnides目录处于同一级别，因为datavacuum不仅具有特定于 webcrawling 项目的函数和类。

在路径GUI/arachnides/arachnides/spiders/araignees.py中名为araignees.py的蜘蛛文件中，从datavacuum包导入没有问题/问题： from GUI.datavacuum.database import collection_mongo 。

但是当我使用控制台（在 IDE 本身在Turf环境下）在项目中运行scrapy.Spider时，它会返回此错误：

(Turf) avy@machine:~/GUI/arachnides/arachnides/spiders$ scrapy crawl test
Traceback (most recent call last):
  File "/home/avy/anaconda3/envs/Turf/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/cmdline.py", line 144, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 290, in __init__
    super().__init__(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 167, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 161, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 67, in from_settings
    return cls(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 51, in _load_all_spiders
    for module in walk_modules(name):
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/misc.py", line 88, in walk_modules
    submod = import_module(fullpath)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/avy/GUI/arachnides/arachnides/spiders/araignees.py", line 13, in <module>
    from GUI.datavacuum.database import collection_mongo
ModuleNotFoundError: No module named 'GUI'

考虑到我在每个目录中都插入了__init__.py文件，我不明白为什么它不起作用，而且考虑到 IDE 和控制台使用的是相同的 python 环境。
使用sys.path.append(absolute_path_of_spiders_folder)添加绝对路径没有任何改变。

其他尝试和结果

将datavacuum目录作为石膏移动

在第一个 arachnides 目录（scrapy 项目的顶层）中移动 datavacuum。

GUI (dir)
 ├── __init__.py (file)
 ├── arachnides (dir) # a scrapy project
 │   ├── __init__.py (file)
 │   ├── scrapy.cfg (file)
 │   ├── .scrapy (dir)
 │   ├── datavacuum (dir) #TO THERE
 │   │    ├── __init__.py (file)
 │   │    ├── database.py (file)
 │   │    └── parse.py (file)
 │   └── arachnides (dir)
 │        ├── __init__.py (file)
 │        ├── items.py (file)
 │        └── spiders (dir)
 │              ├── __init__.py (file)
 │              └── araignees.py (file)
 └── #MOVED FROM HERE

包在araignees.py中的导入

from ... import datavacuum IDE 中没有提到任何问题。

scrapy crawl命令的结果

(Turf) avy@machine:~/GUI/arachnides$ scrapy crawl test
Traceback (most recent call last):
  File "/home/avy/anaconda3/envs/Turf/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/cmdline.py", line 144, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 290, in __init__
    super().__init__(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 167, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/crawler.py", line 161, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 67, in from_settings
    return cls(settings)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spiderloader.py", line 51, in _load_all_spiders
    for module in walk_modules(name):
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/misc.py", line 88, in walk_modules
    submod = import_module(fullpath)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/avy/GUI/arachnides/arachnides/spiders/araignees.py", line 13, in <module>
    from ... import datavacuum
ImportError: attempted relative import beyond top-level package

在第二个蜘蛛目录中移动 datavacuum

GUI (dir)
 ├── __init__.py (file)
 └── arachnides (dir) # a scrapy project
     ├── __init__.py (file)
     ├── scrapy.cfg (file)
     ├── .scrapy (dir)
     ├── #MOVED FROM HERE
     └── arachnides (dir)
          ├── __init__.py (file)
          ├── items.py (file)
          ├── datavacuum (dir) #TO THERE
          │    ├── __init__.py (file)
          │    ├── database.py (file)
          │    └── parse.py (file)
          └── spiders (dir)
                ├── __init__.py (file)
                └── araignees.py (file)

包在araignees.py中的导入

from .. import datavacuum IDE 中没有提到任何问题。
在蜘蛛测试本身（在spiders.py中）有这个功能用于检查：
datavacuum.parse.bidule() 。
datavacuum package中的功能bidule ：

def bidule():
    print("I print to check")

scrapy crawl命令的结果

(Turf) avy@machine:~/GUI/arachnides$ scrapy crawl test
2022-07-02 13:33:17 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: arachnides) #there is just this improvement
...
    Traceback (most recent call last):
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/defer.py", line 132, in iter_errback
    yield next(it)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/python.py", line 354, in __next__
    return next(self.data)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/python.py", line 354, in __next__
    return next(self.data)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
    for x in result:
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spidermiddlewares/referer.py", line 342, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spidermiddlewares/urllength.py", line 40, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 66, in _evaluate_iterable
    for r in iterable:
  File "/home/avy/GUI/arachnides/arachnides/spiders/araignees.py", line 108, in parse
    datavacuum.parse.bidule()
AttributeError: module 'arachnides.datavacuum' has no attribute 'parse'

我什至不明白为什么它在这是一个模块时搜索“属性”。

在蜘蛛目录中移动 datavacuum

GUI (dir)
 ├── __init__.py (file)
 └── arachnides (dir) # a scrapy project
     ├── __init__.py (file)
     ├── scrapy.cfg (file)
     ├── .scrapy (dir)
     └── arachnides (dir)
          ├── __init__.py (file)
          ├── items.py (file)
          ├── #MOVED FROM HERE
          └── spiders (dir)
                ├── __init__.py (file)
                ├── araignees.py (file)
                └── datavacuum (dir) #TO THERE
                     ├── __init__.py (file)
                     ├── database.py (file)
                     └── parse.py (file)

包在araignees.py中的导入

from . import datavacuum from . import datavacuum IDE 中没有提到任何问题。
但请注意，IDE 突出显示了parse for datavacuum.parse.bidule() “Cannot find reference 'parse' in ' init .py'”，为什么会这样？

scrapy crawl命令的结果

(Turf) avy@machine:~/GUI/arachnides$ scrapy crawl test
2022-07-02 14:00:25 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: arachnides)
...
I print to check #datavacuum.parse.bidule() does work well
...
2022-07-02 14:03:12 [scrapy.core.engine] INFO: Spider closed (finished)

像这样它可以工作，但是当您查看整个问题时，它似乎不稳定。

在GUI根目录中创建 setup.py

正如这个流行的答案中提到的，我做了一个setup.py如下：

GUI (dir)
 ├── setup.py (file) #SETUP HERE    
 ├── __init__.py (file)
 ├── arachnides (dir) # a scrapy project
 │   ├── __init__.py (file)
 │   ├── scrapy.cfg (file)
 │   ├── .scrapy (dir)
 │   └── arachnides (dir)
 │        ├── __init__.py (file)
 │        ├── items.py (file)
 │        ├── settings.py (file)
 │        ├── pipelines.py (file)
 │        ├── middlewares.py (file)
 │        └── spiders (dir)
 │              ├── __init__.py (file)
 │              └── araignees.py (file)
 │ 
 ├── datavacuum (dir)
 │    ├── __init__.py (file)
 │    ├── database.py (file)
 │    └── parse.py (file)
 │
 └── GUI.egg-info (dir) # done after the command pip install -e .
      ├── dependency_links.txt (file) #empty
      ├── PKG-INFO (file)
      ├── top_level.txt (file)
      └── SOURCES.txt (file)

pip install -e .

(Turf) avy@machine:~/GUI$ pip install -e .
Obtaining file:///home/GUI
Installing collected packages: GUI
  Attempting uninstall: GUI
    Found existing installation: GUI 1.0
    Uninstalling GUI-1.0:
      Successfully uninstalled GUI-1.0
  Running setup.py develop for GUI
Successfully installed GUI-1.0

在SOURCES.txt和top_level.txt

#SOURCES.txt
setup.py
GUI.egg-info/PKG-INFO
GUI.egg-info/SOURCES.txt
GUI.egg-info/dependency_links.txt
GUI.egg-info/top_level.txt
arachnides/__init__.py
arachnides/arachnides/__init__.py
arachnides/arachnides/items.py
arachnides/arachnides/middlewares.py
arachnides/arachnides/pipelines.py
arachnides/arachnides/settings.py
arachnides/arachnides/spiders/__init__.py
arachnides/arachnides/spiders/araignees.py
datavacuum/__init__.py
datavacuum/database.py
datavacuum/parse.py

#top_level.txt
arachnides
datavacuum

项目中爬取爬虫时的结果

(Turf) avy@machine:~/GUI/arachnides$ scrapy crawl test
Traceback (most recent call last):
  File "/home/avy/anaconda3/envs/Turf/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/cmdline.py", line 114, in execute
    settings = get_project_settings()
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/utils/project.py", line 68, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/site-packages/scrapy/settings/__init__.py", line 287, in setmodule
    module = import_module(module)
  File "/home/avy/anaconda3/envs/Turf/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'arachnides.settings'

......总是有理由犯错......

带有sys的 Python 路径

(base) avy@machine:~/GUI/arachnides$ python
Python 3.9.12 (main, Apr  5 2022, 06:56:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/avy/anaconda3/lib/python39.zip', '/home/avy/anaconda3/lib/python3.9', '/home/avy/anaconda3/lib/python3.9/lib-dynload', '/home/avy/anaconda3/lib/python3.9/site-packages']
>>> from GUI import datavacuum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'GUI'
>>> sys.path.insert(0,"/home/avy/GUI/datavacuum")
>>> from GUI import datavacuum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'GUI'
>>> sys.path.insert(0,"/home/avy/GUI")
>>> from GUI import datavacuum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'GUI'
>>>

即使在基础环境中，模块导入也存在严重问题

0 个解决方案

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 将 Spark avro 包导入到 dockerized python 项目中以在 S3 中导入 avro 文件 导入外部包的正确约定是什么？ 点子和导入有麻烦 在Scrapy项目中使用Django的模型（在管道中） 在同一个项目的不同子目录中导入python模块和包 将标头和有效负载导入Scrapy Scrapy-导入JSON时出错 在 Python 中导入包 导入winpython软件包的问题 在虚拟环境中导入包
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM