Scrapy：努力实现爬行蜘蛛

Question

I've been trying to implement a web crawler to scrape titles and points off hacker news website. I had success with parsing it through using the normal scrapy.spider class. However, I'd like to have a robust way of crawling through links using link extractor. Here's my current setup:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class QuotesSpider(CrawlSpider):
    name = "crawl"
    allowed_domains = ['news.ycombinator.com']

    start_urls = [
        'https://news.ycombinator.com/news?p=2',
    ]


    rules = [
    Rule(LinkExtractor(allow=r'news?p=[3-9]'), callback='parse_news', follow=True)
]
 
    def parse_news(self, response):

        data = {}
        title = response.xpath("//td/a[@class='storylink']/text()").getall()
        point = response.xpath("//td[@class='subtext']/span/text()").getall()
        length = len(title)

        for each in range(length):
            data["title"] = title[each]
            data["point"] = point[each]
            yield data

I can't seem to get any information saved to a json after running this though.

Answer 1

Your code has a lot of errors, but for the first step, you have to fix the LinkExtractor:

Rule(LinkExtractor(allow=r'news\?p=[3-9]'), callback='parse_news', follow=True)

The question mark is an especial character in regular expressions, so you have to put a \\ before it. Next, you have to fix the data extraction process in your for loop.