Scrapy在parse中没有yield request后直接停止

Question

I'm trying to make a spider that goes through a certain amount of start urls and if the resulting page is the right one I yield another request. The problem is that if I try anyway of not yielding a second request the spider will stop directly. There are no problems if I yield the second request.

Here is the relevant code:

def start_requests(self):
        urls = ['https://www.hltv.org' + player for player in self.hashPlayers]
        print(len(urls))
        for url in urls:
            return [scrapy.Request(url=url, callback=self.parse)]

    def parse(self, response):
        result = response.xpath("//div[@class = 'playerTeam']//a/@href").get()
        if result is None:
            result = response.xpath("//span[contains(concat(' ',normalize-space(@class),' '),' profile-player-stat-value bold ')]//a/@href").get()

        if result is not None:
            yield scrapy.Request(
                url = "https://www.hltv.org" + result,
                callback = self.parseTeam
            )

So I want a way to make the spider to continue after I call the parse function and don't yield a request.

Answer 1

def start_requests(self):
    urls = ['https://www.hltv.org' + player for player in self.hashPlayers]
    print(len(urls))
    for url in urls:
        return [scrapy.Request(url=url, callback=self.parse)]

If you use return , the function is terminated, the loop won't iterate to the next value and a single request will be sent to the Scrapy Engine. Replace it with yield so it returns a generator.