Python Scrapy：返回抓取的 URL 列表

Question

I am using scrapy to scrape all the links off single domain. I am following all links on the domain but saving all links off the domain. The following scraper works correctly, but I can't access member variables from within the scraper since I am running it with a CrawlerProcess .

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['https://example.com']

    on_domain_urls = set()
    off_domain_urls = set()

    def parse(self, response):
        links = response.xpath('//a/@href')
        for link in links:
            url = link.get()
            if 'example.com' in url and url not in self.on_domain_urls:
                print('On domain links found: {}'.format(
                    len(self.on_domain_urls)))
                self.on_domain_urls.add(url)
                yield scrapy.Request(url, callback=self.parse)
            elif url not in self.off_domain_urls:
                print('Offf domain links found: {}'.format(
                    len(self.on_domain_urls)))
                self.off_domain_urls.add(url)

process = CrawlerProcess()
process.crawl(GoodOnYouSpider)
process.start()
# Need access to off_domain_links

How can I access off_domain_links? I could probably move it to a global scope but this seems hack. I can also append to a file, but I'd like to avoid file I/O if possible. Is there a better way to return aggregated data like this?

Answer 1

Did you check the Itempipeline? I think you'll have to use that in this scenario and decide what needs to be done with the variable.

See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html