Scrapy搜寻器遵循包含关键字的链接

Question

I have a scrapy webcrawler that is working really well. However, I would like to make it follow only links that contain a certain keyword or phrase. I thought I had it figured out, but my output was not correct.

from scrapy.selector import HtmlXPathSelector
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.utils.url import urljoin_rfc
from webcrawler.items import SitegraphItem


class GraphspiderSpider(CrawlSpider):
    name = "examplespider"
    custom_settings = {
    'DEPTH_LIMIT': '2',
    }
    allowed_domains = []
    start_urls = (
        'http://www.example.com/products/',
    )

    rules = (
        Rule(LinkExtractor(allow=r'/'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        hxs = HtmlXPathSelector(response)
        i = SitegraphItem()
        i['url'] = response.url
        # i['http_status'] = response.status
        llinks=[]
        for anchor in hxs.select('//a[text()="keyword"]/@href'):
            href=anchor.select('@href').extract()[0]
            if not href.lower().startswith("javascript"):
                llinks.append(urljoin_rfc(response.url,href))
        i['linkedurls'] = llinks
        return i

    def _response_downloaded(self, response):
        filename = response.url.split("/")[-1] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.body)    
        rule = self._rules[response.meta['rule']]
        return self._parse_response(response, rule.callback, rule.cb_kwargs, rule.follow)

I added the "keyword" statement in the hxs.select segment but that clearly was not correct. I'm not sure how to get the keyword to work right.

Answer 1

See if you can implement your link-filtering logic using LinkExtractor attributes.

Otherwise, use Spider instead of CrawlSpider . CrawlSpider is only useful for the limited use cases it supports; Spider works for all use cases.