python scrapy 从产品页面获取 url 列表

Question

I am trying to scrape this page: https://www.jomashop.com/watches.html?dir=desc&order=bestsellers

my current code to get a list of the product urls is:

def start_requests(self):
    yield scrapy.Request(
        url=self.start_url,
        callback=self.parse_product_url,
        headers=self.headers,
        dont_filter=True,
        meta={"page": 1}
        )

def parse_product_url(self, response):
    if response.status in self.handle_httpstatus_list:
        time.sleep(5)
        yield response.request
    else:
        meta_product = response.meta
        meta_product["rank"] = 0
        meta_product["category_url"] = response.url
        product_urls = response.xpath("//a[@class='productName-link']/a/@href").extract()

but the product_urls is consistently returning empty. I am not well versed in the syntax of xpath, so I think there is a problem with how I have worded it. Any help to get the product url is appreciated! thank you!

I have also tried:

response.xpath("//div[@class='product-details']/h2[@class='productName-link']/text()")
response.xpath("//a[@class='productName-link']/a/@href").extract()
response.xpath("//*[contains(@class, 'product-details')]/a/@href").extract()
response.xpath("//*[contains(@class, 'productName-link')]/a/@href").extract()
response.xpath("//*[@id='product-list-wrapper']/ul/li[4]/div/div[2]/h2/a").extract()
response.xpath("/html/body/div[1]/div/main/div[1]/div[2]/div[2]/div[2]/ul/li[4]/div/div[2]/h2/a").extract()

I feel like I am so close, but none of these seem to be working.

Answer 1

It doesn't look like that link provides any links by that class in the source (the source is the data that comes from the web server to the client, which in this case is your python script). If you look at the page source, it's mainly just JavaScript. But if you inspect the DOM from within a browser, you see the links you are looking for, meaning the page uses JavaScript (most likely) to populate the DOM with the elements you are looking for. This is done because the browser runs the JavaScript which populates the DOM. Scrapy does not execute any JavaScript and all it has access to is the data sent initially by the Web Server. Scrapy's documentation refers to this as dynamically-loaded content and provides this suggestion:

Selecting dynamically-loaded content Some webpages show the desired data when you load them in a web browser. However, when you download them using Scrapy, you cannot reach the desired data using selectors. When this happens, the recommended approach is to find the data source and extract the data from it.