自定义python包中的导入错误

Question

While reorganizing a scrapy project I want to create a package from my pipelines.py file.

I want to change this...

├── my_scraper
│   ├── __init__.py
│   ├── items.py
│   ├── pipelines.py
│   ├── settings.py
│   └── spiders
└── scrapy.cfg

to this...

├── my_scraper
    │   ├── __init__.py
    │   ├── items.py
    │   ├── pipelines
    │   │   ├── __init__.py
    │   │   ├── MyPipeline1.py
    │   │   ├── MyPipeline2.py
    │   │   └── MyPipeline3.py
    │   ├── settings.py
    │   └── spiders
    └── scrapy.cfg

But after importing each pipeline class in pipelines/__init__.py

from my_pipeline1 import MyPipeline1
from my_pipeline2 import MyPipeline2
from my_pipeline3 import MyPipeline3

in and referencing each in settings.py

ITEM_PIPELINES = {
'my_scraper.pipelines.MyPipeline1': 100,
'my_scraper.pipelines.MyPipeline2': 200,
'my_scraper.pipelines.MyPipeline3': 300,

}

Now when I run scrapy crawl my_scraper I get the error...

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
      File "<frozen importlib._bootstrap>", line 969, in _find_and_load
      File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 665, in exec_module
      File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
      File "/Users/path/to/my_scraper/pipelines/__init__.py", line 1, in <module>
        from my_pipeline1 import MyPipeline1
    ImportError: No module named 'my_pipeline1'

Not sure how I should approach this after in order to make my code maintainable in the long run. Any tips / help are appreciated!

Answer 1

But after importing each pipeline class in pipelines/ init .py

from my_pipeline1 import MyPipeline1
from my_pipeline2 import MyPipeline2
from my_pipeline3 import MyPipeline3

Looking at your project tree it seems like the above is completely unnecessary, in fact you don't even have anything named my_pipeline1 .

So just get rid of these lines in pipelines/__init__.py .
You can nest and have multiple packages in your package. In this case you have my_spider package which in itself contains spiders and pipelines packages, so you can safely import them with:

from myspider.pipelines.pipelines1 import SomeClass
# or in settings as
ITEM_PIPELINES = {
    'myspider.pipelines.pipelines1.MyPipeline': 100,
}