在scrapy中将单个蜘蛛解耦为不同的蜘蛛

Question

I'd like to decouple parsing into a different spiders.

Currently I have:

class CategoriesSpider(scrapy.Spider):
    name = 'categories'
    allowed_domains = ['example.org']
    start_urls = ['https://example.org/categories']

    def parse(self, response):
        for link in response.xpath("//div[@class='something']/@href"):
          yield scrapy.Request(response.urljoin(link.root), callback=self.parse_actual_item_i_want)

    def parse_actual_item_i_want(self, response):
          yield self.find_the_item(response)

And now split it into:

class OneThingSpider(scrapy.Spider):
    name = 'one_thing'
    allowed_domains = ['example.org']
    start_urls = ['https://example.org/']

    def __init__(self, url: str):
        if url == None or url == "":
            raise ValueError("Invalid url given")

        # Exact URL and it's format is not known
        self.start_urls = [url]

    def parse(self, response):
          yield self.find_the_item(response)

So that if only one thing is updated I can only use OneThingSpider

So where and how I call the OneThingSpider inside CategoriesSpider or pipeline?

I've tried these:

Inside the CategoriesSpider attempt #1:

    def parse(self, response):
        for link in response.xpath("//div[@class='something']/@href"):
          yield self.crawler.crawl("one_thing", response.urljoin(link.root))

In the pipeline attempt #1:

In CategoriesSpider :

    def parse(self, response):
        for link in response.xpath("//div[@class='something']/@href"):
          yield CategoryUrl({"url": response.urljoin(link.root)})

In pipeline:

class MyPipeline(object):
    def process_item(self, item, spider):
        if isinstance(item, CategoryUrl):
            spider.crawler.crawl("one_thing", item["url"])
            return

        # this part works
        if isinstance(item, ItemIWant): 
            with open(...)

In the pipeline attempt #2:

Same change in the CategoriesSpider as in attempt #1. Here we simply are trying to push the URLs to the OneThingSpider after CategoriesSpider has closed.

In pipeline:

class MyPipeline(object):
    urls = []
    def process_item(self, item, spider):
        if isinstance(item, CategoryUrl):
            self.urls.append(item["url"])
            spider.crawler.crawl("one_thing", item["url"])
            return

        # this part works
        if isinstance(item, ItemIWant): 
            with open(...)

    def close_spider(self, spider):
        if spider.name == "categories":
            for url in self.urls:
                spider.crawler.crawl("one_thing", url)

The error I get is that the crawler is already crawling . So what is the correct way for decoupling spider into smaller spiders?

Goal is that I can run:

scrapy crawl categories

and also

scrapy crawl one_thing -a url="https://example.org/something/xyz.html"

Answer 1

Usually you'd use class inheritence for reusing spider code in multiple spiders:

class BaseSpider(Spider):
    start_urls = NotImplemented
    start_body = ""

    def start_requests(self):
        for url in self.start_urls:
            yield Request(url, method='POST', body=self.start_body)

    def parse(self, response):
        raise NotImplementedError


# then children spiders
class CakeSpider(BaseSpider):
    name = 'cakes'
    start_urls = ['http://example1.com']
    start_body = '{"category": "cakes"}'

    def parse(self, response):
        # custom parser for first spider here

class VegetableSpider(BaseSpider):
    name = 'vegetables'
    start_urls = ['http://example2.com']
    start_body = '{"category": "vegetables"}'

    def parse(self, response):
        # custom parser for second spider here

As a result you'd run scrapy crawl cakes to start crawling cake category and scrapy crawl vegetables to crawl vegetable category.