scrapy crawlspider没有关注下一个链接

Question

我正抓狂地收集意大利国家警察的新闻稿。 我遇到的问题是，即使我设置了规则来查找“下一个”或意大利语中的“成功”按钮并遵循该链接，刮板也未遵循“下一个”链接。

这是我的代码。

from scrapy.spiders import Rule, CrawlSpider
from scrapy.linkextractors import LinkExtractor
from scrapy.selector import Selector
from items import ScrapyCrimeScraperItem
from string import replace

class ItalyScraper(CrawlSpider):
name = 'italy_crawler_test'
allowed_domains = ['poliziadistato.it']

start_urls = [
    'http://www.poliziadistato.it/archivio/category/1298/2015/',
    'http://www.poliziadistato.it/archivio/category/1298/2015/9/'

]

rules = (Rule(LinkExtractor(allow=('http://.*/articolo/view/*.....')), callback='parse_article', follow=False),
Rule(LinkExtractor(restrict_xpaths=("/html/body/div[@class='container'[1]/div[@class='row']/div[@class='col-md-6 col-md-push-3 padding0']/div[@class='trecolonne']/div[@class='center']/div[@class='bar']/ul[@class='paginazione']/li/a[contains(""@title,""'Successiva')]",)), follow=True))

# def generate_article_links(self, response):
#     for href in response.css('a'):
#         url = href.extract()
#         yield scrapy.Request(url, callback=self.parse_article)


def parse_article(self, response):
    self.logger.info('Hi, this is an item page! %s', response.url)
    item = ScrapyCrimeScraperItem()
    item['city'] = response.selector.css('h1').extract()[0]
    item['country'] = 'italy'
    item['site_link'] = response.url
    item['article_link'] = response.url
    item['article_raw_text'] = self.remove_carriage_returns(response.selector.css('.resetfont '
                                                     'p').extract(

    )[0])
    item['article_raw_date'] = response.selector.css('.data').extract()[0]
    item['article_translated_text'] = ''
    item['article_translated_date'] = ''
    item['article_raw_markup'] = ''
    item['crimes'] = ''
    item['locations'] = ''
    item['dateformat'] = ''
    item['reserved1'] = ''
    item['reserved2'] = ''

    yield item


def remove_carriage_returns(self,item):

    return(item.replace("\n", " "))

我浏览了对类似问题的其他答复，但是我在第二条规则上使用了显式的follow=True 。 我需要回调来生成新请求吗？或者follow变量不应该负责生成新请求吗？

Answer 1

我认为您只是搞砸了XPath表达式中的引号。 改用一个简单的方法：

//ul[@class="paginazione"]/li/a[contains(@title, "Successiva")]