Scrapy web 抓取中的 AttributeError

Question

我寫了一個 scrapy 代碼來抓取網站但出現屬性錯誤。 我是 web 抓取的新手，請指導我如何解決此錯誤。 這是錯誤消息： AttributeError: 'str' object has no attribute 'xpath'

這是我的代碼：

# -*- coding: utf-8 -*-
import scrapy


class ShopSpider(scrapy.Spider):
    name = 'shop'
    allowed_domains = ['https://www.redbubble.com']
    start_urls = ['https://www.redbubble.com/shop/shower-curtains/']

    def parse(self, response):

        products = response.xpath("//a[@class='styles__link--2sYi3']").get()
        for product in products:
            product_url = product.xpath(".//img[@class='styles__image--2CwxX styles__productImage--3ZNPD styles__rounded--1lyoH styles__fluid--3dxe-']/@src").get()
            title = name = product.xpath(".//div[@class='styles__box--206r9 styles__paddingRight-0--fzRHs']/div[@class='styles__textContainer--1xehi styles__disableLineHeight--3n9Fg styles__nowrap--2Vk3A']/span/text()").get()
            yield {
                'name'  :   title,
                'url'   :   product_url
            }

Answer 1

錯誤很明顯

您正在嘗試從字符串中調用xpath方法

請更換

products = response.xpath("//a[@class='styles__link--2sYi3']").get()

至

products = response.xpath("//a[@class='styles__link--2sYi3']")

Answer 2

這是對我有用的代碼。 您收到 str 錯誤，因為您不能在字符串后使用 response.xpath。 您需要直接在for循環中使用。 這是我使用的代碼。 您也可以刪除允許的域。

   import scrapy


class ShopSpider(scrapy.Spider):
    name = 'shop'
    start_urls = ['https://www.redbubble.com/shop/shower-curtains/']

    def parse(self, response):
         for product in response.xpath("//a[@class='styles__link--2sYi3']"):
            product_url = product.xpath(
                ".//img[@class='styles__image--2CwxX styles__productImage--3ZNPD styles__rounded--1lyoH styles__fluid--3dxe-']/@src").get()
            title = product.xpath(".//div[@class='styles__box--206r9 styles__paddingRight-0--fzRHs']/div[@class='styles__textContainer--1xehi styles__disableLineHeight--3n9Fg styles__nowrap--2Vk3A']/span/text()").get()
            yield {
                "title": title,
                "url": product_url
            }