Unable to import Scrapy Spider into Script

Question

All my imports are working except for importing class SomeSpider into main/main.py from spider/src.py . The spider itself runs when I call scrapy crawl somespider in the terminal. Does python not recognize modules with scrapy.spider ?

My file structure:

/whiskers
-/venv
--/bin
--/include
--/lib
--/whiskers
---/whiskers
----/main
-----/main.py
----/spiders
-----/__init__.py
-----/src.py
----/__init__.py
----/items.py
----/middlewares.py
----/pipelines.py
----/settings.py
---/scrapy.cfg
--/pyvenv.cfg

Putting any of these in main/main.py :

from whiskers.spiders.src import SomeSpider
   
(or) 

from whiskers.whiskers.spiders.src import SomeSpider

(or) 

from whiskers.venv.whiskers.whiskers.spiders.src import SomeSpider

gives error:

ModuleNotFoundError: No module named 'whiskers'

Trying either:

from ..spiders.src import SomeSpider

(or)

from .. import LawSpider

gives error:

ImportError: attempted relative import with no known parent package

In spider/src.py the spider itself is just a basic scrapy.Spider named SomeSpider :

class SomeSpider(scrapy.Spider):

Answer 1

Put your main.py script outside a package folder. A package folder is a folder containing an __init__.py . The idea which Guido van Rossum had with this, is that scripts - the starting points, modules where __name__ == '__main__' - should not reside inside but outside of packages. Packages shall just contain library code which gets imported - into scripts or into other packages.

In this case, the simplest is to move the main.py two levels up:

whiskers/
  venv/
    bin/
    include/
    lib/
    whiskers/
      main.py  <-- put it here
      whiskers/
        __init__.py
        spiders/
          __init__.py
          src.py
        items.py
        middlewares.py
        pipelines.py
        settings.py
      scrapy.cfg
    pyvenv.cfg

Then in the main.py do from whiskers.spiders.src import SomeSpider .