Python 3.8 and Scrapy 2.4.1 "No module named <name of my project>

Question

I met a problem I'm not able to jump through. I use scrapy (2.4.1), python3.8.6 and Pycharm. My project structure is like that:

• project_folder

  • scrapy_project

    • spiders

      • init .py
      • my_spider.py

  • init .py

  • items.py

  • middlewares.py

  • pipeline.py

  • run.py

  • settings.py

when I run my spider from Pycharm using Run - it works. when I run my spider using:
scrapy crawl <spider_name> - it works.

But when I do this from terminal using:

               python3 <path_to_the_script/my_spider.py 

a have error:

ModuleNotFoundError: No module named 'scrapy_project'.

       in my_spider.py I used from scrapy_project.items import <Class_name>Item ]

I used print(sys.path) tho check what is in my PYTHONATHS and I have there path to my directory:

          /Users/user/Desktop/project_folder/scrapy_project' 

I have no idea what to do. I want to start my spider from the schedule in Pythonanywhere but when I run my script over there I have got the same error.

Doy You Guys have got any idea where the problem is?

I will be thankful for every tip.

Answer 1

You need to use CrawlerProcess to run as a regular Python script.

Scrapy Documentation