分页不适用于基本的网络爬虫

Question

import scrapy


class BestBooksSpider(scrapy.Spider):
    name = 'best_books'
    page_num = 2
    allowed_domains = [
        'www.goodreads.com/list/show/1.Best_Books_Ever?page=1']
    start_urls = [
        'https://www.goodreads.com/list/show/1.Best_Books_Ever?page=1']

    def parse(self, response):
        page_num = 2
        for books in response.xpath('//tr'):
            yield {
                'Title': books.css('a.bookTitle span::text').get(),
                'Author': books.css('a.authorName *::text').get(),
                'Rating': books.css('span.minirating::text').get(),
            }

        # this part is not working, won't read past page 1

        next_page = 'https://www.goodreads.com/list/show/1.Best_Books_Ever?page=' + \
            str(BestBooksSpider.page_num)
        if BestBooksSpider.page_num < 3:
            BestBooksSpider.page_num += 1
            yield response.follow(next_page, callback=self.parse)

第一页效果很好，但它不会阅读后续页面。 我从其他教程中尝试了许多不同的代码变体，但均未成功。 我在 scrapy 中没有收到任何错误代码。 Scrapy 刚刚表示它已经完成。

Answer 1

您的allowed_domains看起来确实可能是分页不起作用的原因。
allowed_domains = ['www.goodreads.com/list/show/1.Best_Books_Ever?page=1']应该将您的刮板限制在第一页，所以请提前 go 并删除此行并再次尝试您的蜘蛛。