如何跟踪深度为2的Scrapy链接？

Question

I am writing a scraper which should extract all links from an initial webpage if it has any given keywords in the metadata and if they contain 'htt' in the URL, follow them and repeat the procedure twice, so the depth of the scraping will be 2. This is my code:

from scrapy.spider import Spider
from scrapy import Selector
from socialmedia.items import SocialMediaItem
from scrapy.contrib.spiders import Rule, CrawlSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor

class MySpider(Spider):
    name = 'smm'
    allowed_domains = ['*']
    start_urls = ['http://en.wikipedia.org/wiki/Social_media']
    rules = (
             Rule(SgmlLinkExtractor(allow=()), callback="parse_items", follow= True),
             )
    def parse_items(self, response):
        items = []
        #Define keywords present in metadata to scrap the webpage
        keywords = ['social media','social business','social networking','social marketing','online marketing','social selling',
            'social customer experience management','social cxm','social cem','social crm','google analytics','seo','sem',
            'digital marketing','social media manager','community manager']
        for link in response.xpath("//a"):
            item = SocialMediaItem()
            #Extract webpage keywords 
            metakeywords = link.xpath('//meta[@name="keywords"]').extract()
            #Compare keywords and extract if one of the defined keyboards is present in the metadata
            if (keywords in metaKW for metaKW in metakeywords):
                    item['SourceTitle'] = link.xpath('/html/head/title').extract()
                    item['TargetTitle'] = link.xpath('text()').extract()
                    item['link'] = link.xpath('@href').extract()
                    outbound = str(link.xpath('@href').extract())
                    if 'http' in outbound:
                        items.append(item)
        return items

But I get this error:

    Traceback (most recent call last):
      File "C:\Anaconda\lib\site-packages\twisted\internet\base.py", line 1201, in mainLoop
        self.runUntilCurrent()
      File "C:\Anaconda\lib\site-packages\twisted\internet\base.py", line 824, in runUntilCurrent
        call.func(*call.args, **call.kw)
      File "C:\Anaconda\lib\site-packages\twisted\internet\defer.py", line 382, in callback
        self._startRunCallbacks(result)
      File "C:\Anaconda\lib\site-packages\twisted\internet\defer.py", line 490, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "C:\Anaconda\lib\site-packages\twisted\internet\defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "C:\Anaconda\lib\site-packages\scrapy\spider.py", line 56, in parse
        raise NotImplementedError
    exceptions.NotImplementedError: 

Can you help me to follow the links containing http in its URL? Thanks!

Dani

Answer 1

It is ignoring the rule for two main reasons here:

• you need to use CrawlSpider , not a regular Spider
• specified callback parse_items() doesn't exist. Rename parse() to parse_items() .

Answer 2

在您的代码中，将class MySpider(Spider):更改为class Myspider(crawlSpider) 。