如何反转 Python 中的列表顺序并在返回 None 时停止收益？

Question

I am generating pagination links which I suspect exists with Python 3.x:

start_urls = [
    'https://...',
    'https://...' # list full of URLs
]

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(
            url = url,
            meta={'handle_httpstatus_list': [301]},
            callback=self.parse,
        )

def parse(self, response):
    for i in range(1, 6):
        url = response.url + '&pn='+str(i)
        yield scrapy.Request(url, self.parse_item)

def parse_item(self, response):

        # check if no results page
        if response.xpath('//*[@id="searchList"]/div[1]').extract_first():
            self.logger.info('No results found on  %s', response.url)
            return None
        ...

Those URLs will be processed by scrapy in parse_item. Now there are 2 problems:

1. The order is reverse and I do not understand why. It will request pagen umbers: 5,4,3,2,1 instead of 1,2,3,4,5

2. If the no results are found on page 1, the entire series could be stoped. Parse Item returns already "None", but the I guess I need to adapt the method "parse" to exit the for loop and continue. How?

Answer 1

The scrapy.Request you generate are running in parallel - In other words, there is guarantee for the order how you get the response as it depends on the server.

If some of the requests, depends on the response of of a request, you should yield those requests in its parse callback.

For example:

def parse(self, response):
    url = response.url + '&pn='+str(1)
    yield scrapy.Request(url, self.parse_item, cb_kwargs=dict(page=1, base_url=response.url))
                             

def parse_item(self, response,page, base_url):
        # check if no results page
        if response.xpath('//*[@id="searchList"]/div[1]').extract_first():
            if page < 6:
                url = base_url + '&pn='+str(page+1)
                yield scrapy.Request(url, self.parse_item, cb_kwargs=dict(base_url=base_url,page=page+1))
        else:
            # your code
            yield ...