简体   繁体   中英

why isn't my python script picking up certain variables while web scraping?

I am hoping to get some clarification on this piece of code. When I run the code below the csv output only displays headers "title" and "url". There is no data for "price", "location", "type", or "attributes".

I am just trying to get better at coding through web scraping so I used the real estate website in the code to do so. Thanks in advance for your help!

import requests
from bs4 import BeautifulSoup
from csv import writer

response = requests.get("https://www.rew.ca/properties/areas/kelowna-bc")
soup = BeautifulSoup(response.text, "html.parser")
listings = soup.find_all("article")

with open("property.csv", "w") as csv_file:
    csv_writer = writer(csv_file)
    csv_writer.writerow(["title", "type", "price", "location", "attributes", "link"])
    for listing in listings:
        location = listing.find(class_="displaypanel-info").get_text()
        price = listing.find(class_="displaypanel-title hidden-xs").get_text()
        url = listing.find("a").get('href')
        title = listing.find("a").get('title')
        type = listing.find(class_="displaypanel-info").get_text()
        attributes = listing.find(class_="displaypanel-section clearfix").get_text()
        csv_writer.writerow([title, type, price, location, attributes, url])

use the strip function to remove redundant spaces

for listing in listings:
    location = listing.find(class_="displaypanel-info").get_text().strip()
    price = listing.find(class_="displaypanel-title hidden-xs").get_text().strip()
    url = listing.find("a").get('href').strip()
    title = listing.find("a").get('title').strip()
    type = listing.find(class_="displaypanel-info").get_text().strip()
    attributes = listing.find(class_="displaypanel-section clearfix").get_text().strip()
    csv_writer.writerow([title, type, price, location, attributes, url])`enter code here`

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM