Scrapy: cannot load items to spider

Question

I cannot load scrapy items to scrapy spiders. Here is my project structure:

-rosen
  .log
  .scrapers
    ..scrapers
      ...spiders
        ....__init__.py
        ....exampleSpider.py
      ...__init__.py
      ...items.py
      ...middlewares.py
      ...pipelines.py
      ...settings.py
  .src
    ..__init__.py
    ..otherStuff.py
  .tmp

This structure has been created using scrapy startproject scrapers inside of rosen project (directory).

Now, the items.py has the following code:

import scrapy
from Decimal import Decimal

class someItem(scrapy.Item):
   title: str = scrapy.Field(serializer=str)
   bid: Decimal = scrapy.Field(serializer=Decimal)

And the exampleSpider.py has the following code:

import scrapy
from __future__ import absolute_import

from scrapy.loader import ItemLoader
from scrapers.scrapers.items import someItem

class someSpider(scrapy.Spider):
   name = "some"

   def __init__(self, **kwargs):
       super().__init__(**kwargs)
       self._some_fields = someItem()

   def parse(self, response) -> None: 
       some_loader = ItemLoader(item=self._some_fields, response=response)
       print(self._some_fields.keys())

The error I get is the following: runspider: error: Unable to load 'someSpider.py': No module named 'scrapers.scrapers'

I found Scrapy: ImportError: No module named items and tried all three solutions by renaming and adding from __future__ import absolute_import . Nothing helps. Please advice.

The command that I execute is scrapy runspider exampleSpider.py . I tried it from the spiders and rosen directories.

Answer 1

i do not see any virtualenv inside your directory. So i recommend you to do so eg. under 'rosen'. you can try this:

try:
    from scrapers.items import someItem
except FileNotFoundError:
    from scrapers. scrapers.items import someItem

then cal it with:

scrapy crawl NameOfSpider

or:

scrapy runspider path/to/spider.py