如何确定从`yield scrapy.Request`返回的生成器是否有任何数据？

Question

In the Scrapy Tutorial , the spider extracts the next page links from class="next" and crawls them -

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

For my case, I can't find the next page links in the files downloaded from the webserver but I know the format is response.url concatenated with /page/[page number]/ . Requested pages which don't yield quotes still return a response , for example - No quotes found! . Since the number of next pages is normally less than 20, I could loop over all the possible urls by replacing last 3 lines of the spider with -

for page_num in range(2, 20):
    yield response.follow(f"/page/{page_num}/", callback=self.parse)

However this forces the spider to request pages (such as http://quotes.toscrape.com/page/11 to 20) which don't yield quotes. How can I adjust my spider to terminate the page_num loop after requesting the first page which does not yield quotes? (such as http://quotes.toscrape.com/page/11 )

pseudo code -

    page_num = 2
    while (quotes are yielded from the response):
        yield response.follow(f"/page/{page_num}/", callback=self.parse)
        page_num += 1

Answer 1

You can use result of response.css('..') as condition to next page.
In this case Your code will be like this:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        page_num = get_pagenumber_from_url(response.url)
        
        quotes_sel = response.css('div.quote'):
        # quotes_sel - will be SelectorList if page have item data
        # quotes_sel - will be None if page doesn't have item data
        for quote in quotes_sel:
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        if quotes_sel:
            next_page_url = f"/page/{str(page_num+1)}"
            yield response.follow(next_page_url , callback=self.parse)