如何修复 Scrapy 深度爬行不起作用

Question

I'm currently trying to create a small web scraping prototype using scrapy. My current issue is related to link extraction and following.

I'm trying to make scrapy explore pages and find links to pages (not images and other content for now) but I don't know how to parameter it correctly.

This is the spider I'm using :

class DefaultSpider(CrawlSpider):

    name = "default"
    session_id = -1
    rules = [Rule(LinkExtractor(allow=()),callback='parse', follow=True)]
        
def start_requests(self):
     #not relevent code that gives an URL list to be scrawled
     for url in listurl:

     #make scrapy follow only the current domain url.
     self.rules[0].allow=url
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
     page = Website(response.url,response.text)
     DBInterface.store(page)

The spider doesnt seem to find any links in the pages. I think I'm not doing it the proper way. I tried to another function as a callback instead of the parse method. (changing the rule callback parameter too)

def processlinks (self,response)
    page = Website(response.url,response.text)
    DBInterface.store(page)

edit: update code + title for proper understanding..

Answer 1

CrawlSpider is a special kind of spider that adds rules support to follow links (not extract them btw).

For this spider to work, you can't override the start_requests and parse methods

About getting links, I would recommend using the LinkExtractor which makes the extraction cleaner:

from scrapy

def find_links(self, response):
    for link in LinkExtractor().extract_links(response):
        logging.info('Extracting new url' + link.url)
        yield scrapy.Request(link.url, callback=self.insert_linkDB)

More and updated information for LinkExtractor is available in the documentation

Answer 2

It is somewhat tricky to make CrawlSpider handle initial URLs same way as those that it subsequently extracts with LinkExtractor , which is what you want here. The problem is that you should not define custom callback for any requests that you manually initiate, as it would prevent LinkExtractor from working. On the other hand, you want to perform some action for each crawled URL, including the initial URLs. For those URLs extracted with LinkExtractor , you can provide the callback when defining the rule, but that obviously wouldn't work for initial URLs which are not extracted using these rules. For this purpose, Scrapy provides another method parse_start_url(response) which you can, and should override. So in your case, the following would do what you want:

class DefaultSpider(CrawlSpider):

    name = "default"
    session_id = -1
    rules = [Rule(LinkExtractor(allow=()),callback='parse_results', follow=True)]

def start_requests(self):
     #not relevent code that gives an URL list to be scrawled
     for url in listurl:

     #make scrappy follow only the current domain url.
     self.rules[0].allow=url
        yield scrapy.Request(url=url)

def parse_start_url(self, response):
     self.parse_results(response)

def parse_results(self, response):
     page = Website(response.url,response.text)
     DBInterface.store(page)