无法从scrapy中的所有页面获取数据

Question

I am unable to fetch all pages using below code it only gives data upto page 90 and then show arribute error. I am using next button url to move to the next page. But after page 90 it is giving error that i have mentioned below.

Running this code:

import scrapy
import re

class PaginationSpider(scrapy.Spider):
    name = 'pagination'
    allowed_domains = ['www.farfetch.com']
    start_urls = ['https://www.farfetch.com/de/shopping/men/shoes-2/items.aspx?page=1']

    total_pages_pattern = r'"totalPages":(\d+)'
    current_page_pattern = r"page=(\d+)"

    def parse(self, response):
        
        number_of_pages= int(re.search(self.total_pages_pattern, str(response.body)).group(1))
        current_page = int(re.search(self.current_page_pattern, response.url).group(1))
        
        for brand in response.xpath("//h3[@itemprop='brand']//text()"):

            yield {
                "brand":brand.get()
            }

        if current_page <= number_of_pages:

            next_page = "https://www.farfetch.com/de/shopping/men/shoes-2/items.aspx?page=" + str(current_page+1)
            
            print("Current_page:" + str(current_page))

            yield response.follow(url=next_page, callback=self.parse)

Error :

Answer 1

    current_page = int(re.search(self.current_page_pattern, response.url).group(1))

re.search() method will return a Re object if the pattern matches the string. If there is no match, it will return None . So, when the pattern doesn't match, you are calling .group(1) in None .

That's why you are getting an AttributeError .

I didn't execute you code, but you can probably solve it by adding a if statement.