没有使用xpath和Scrapy从div类中获取所有a元素

Question

I have been trying to get all the properties from this website. When I access all of them on the main search page I can retrieve all the information from all the properties, however when I need the information from actual property link, it only seems to go through one property link.

The main issue is in the link part, so when I actually try to access the link of the property. I only get the link and information from the first property but not from all the others.

class PropDataSpider(scrapy.Spider):
    name = "remax"
    start_urls = ['https://www.remax.co.za/property_search/for-sale/?minprice=100000&maxprice=1000000000&displayorder=date&cities=432']


    def parse(self, response):

        propertes = response.xpath("//div[@class='w-container main-content remodal-bg']")
        for prop in propertes:
            link = 'http://www.remax.co.za/' + prop.xpath("./a/@href").extract_first()
            agency = self.name
            title = prop.xpath(
                ".//div[@class='property-item']/div[@class='w-clearfix']/p[@class='property-type']/text()").extract_first().strip()
            price = prop.xpath(
                 ".//div[@class='property-item']/div[@class='w-clearfix']/div/strong/text()").extract_first().strip()

...


           yield scrapy.Request(
                link,
                callback=self.parse_property,
                meta={
                    'agency': agency,
                    'title': title,
                    'price': price,
                    'description': description,
                    'bedrooms': bedrooms,
                    'bathrooms': bathrooms,
                    'garages': garages,
                }
            )


 def parse_property(self, response):
        agency = response.meta["agency"]
        title = response.meta["title"]
        price = response.meta["price"]
        description = response.meta["description"]
        bedrooms = response.meta["bedrooms"]
        bathrooms = response.meta["bathrooms"]
        garages = response.meta["garages"]


        yield {'agency': agency, 'title': title, 'price': price, "description": description, 'bedrooms': bedrooms,'bathrooms': bathrooms, 'garages': garages}

What I would like to get is all the other links to properties. I am not sure what I am doing wrong and how to fix this.

Thank you very much for help!

Answer 1

You need couple of changes:

properties = response.xpath("//div[@class='w-container main-content remodal-bg']/a")
for prop in properties:
    link = 'http://www.remax.co.za/' + prop.xpath("./@href").extract_first()