Find nested div/class with Beautiful Soup

Question

I'm looking at this URL: https://www.zillow.com/homedetails/2616-Tinmouth-St-Austin-TX-78748/64743359_zpid/

My goal is to just pull the damn zestimate and throw it in a list.

The class it's in is: class="Text-c11n-8-65-2__sc-aiai24-0 eUxMDw" .

I tried going higher in the html with:

for div in soup.findAll('div', attrs={'id':'.Home-value '}):
    print (div.text)

also tried select:

items = [item.text.strip() for item in soup.select('.ds-home-values.' )]
items

I also tried extracting:

for t in soup.select("pfs-upsell"):
    t.extract()

every time I get an empty list. I'm thinking maybe there's something I don't understand about this type of HTML.

Answer 1

The website is dynamic. Your element selection selects nothing but that's not the big reason too. The main reason behind to get data is that data is in json format under html DOM with comment.

import requests
import re
import json
url='https://www.zillow.com/homes/for_sale/2_p/?searchQueryState=%7B%22pagination%22%3A%7B%22currentPage%22%3A2%7D%2C%22mapBounds%22%3A%7B%22west%22%3A-97.88981437683105%2C%22east%22%3A-97.78407096862793%2C%22south%22%3A30.136145838104586%2C%22north%22%3A30.208863801102932%7D%2C%22isMapVisible%22%3Atrue%2C%22filterState%22%3A%7B%22sort%22%3A%7B%22value%22%3A%22days%22%7D%2C%22ah%22%3A%7B%22value%22%3Atrue%7D%7D%2C%22isListVisible%22%3Atrue%2C%22mapZoom%22%3A13%7D'
r = requests.get(url,headers = {'User-Agent':'Mozilla/5.0'})

data = json.loads(re.search(r'!--(\{"queryState".*?)-->', r.text).group(1))

for item in data['cat1']['searchResults']['listResults']:
    price=item['unformattedPrice']
    print(price)
    listing_url='https://www.zillow.com'+item['detailUrl']
    print(listing_url)

Output:

470000
https://www.zillow.comhttps://www.zillow.com/homedetails/7337-Menchaca-Rd-36-Austin-TX-78745/119617777_zpid/
539500
https://www.zillow.comhttps://www.zillow.com/homedetails/3214-Barnsley-Dr-Austin-TX-78745/29488852_zpid/
530000
https://www.zillow.comhttps://www.zillow.com/homedetails/10041-Aly-May-Dr-Austin-TX-78748/119624978_zpid/
659000
https://www.zillow.comhttps://www.zillow.com/homedetails/9831-Briar-Ridge-Dr-Austin-TX-78748/29509925_zpid/
499000
https://www.zillow.comhttps://www.zillow.com/homedetails/10209-Brantley-Bnd-Austin-TX-78748/29511458_zpid/
550000
https://www.zillow.comhttps://www.zillow.com/homedetails/7509-Bender-Dr-Austin-TX-78749/29486187_zpid/
585000
https://www.zillow.comhttps://www.zillow.com/homedetails/3012-Cohoba-Dr-Austin-TX-78748/251027808_zpid/
749900
https://www.zillow.comhttps://www.zillow.com/homedetails/8302-Hornet-Dr-Austin-TX-78749/29494207_zpid/
750000
https://www.zillow.comhttps://www.zillow.com/homedetails/7400-Trede-Cv-Austin-TX-78745/29497905_zpid/
630000
https://www.zillow.comhttps://www.zillow.com/homedetails/8610-Coastal-Dr-Austin-TX-78749/29499263_zpid/
815000
https://www.zillow.comhttps://www.zillow.com/homedetails/10800-Harley-Ave-Austin-TX-78748/70332585_zpid/
559900
https://www.zillow.comhttps://www.zillow.com/homedetails/8317-Empress-Blvd-Austin-TX-78745/63440918_zpid/
642000
https://www.zillow.comhttps://www.zillow.com/homedetails/8901-Curlew-Dr-Austin-TX-78748/29504016_zpid/
599000
https://www.zillow.comhttps://www.zillow.com/homedetails/405-Hogg-Pecan-Pass-Austin-TX-78748/80097626_zpid/
479000
https://www.zillow.comhttps://www.zillow.com/homedetails/2505-Jesse-Owens-Dr-Austin-TX-78748/70335498_zpid/
649000
https://www.zillow.comhttps://www.zillow.com/homedetails/2300-Yvette-Cv-Austin-TX-78748/119623072_zpid/
600000
https://www.zillow.comhttps://www.zillow.com/homedetails/8104-Cattle-Dr-Austin-TX-78749/29495648_zpid/
995000
https://www.zillow.comhttps://www.zillow.com/homedetails/3423-Buck-Race-St-Austin-TX-78748/29511556_zpid/
550000
https://www.zillow.comhttps://www.zillow.com/homedetails/7337-Menchaca-Rd-16-Austin-TX-78745/119632513_zpid/
699000
https://www.zillow.comhttps://www.zillow.com/homedetails/2601-Burly-Oak-Dr-Austin-TX-78745/29498040_zpid/
479000
https://www.zillow.comhttps://www.zillow.com/homedetails/2606-Lindsey-Cv-Austin-TX-78748/29506986_zpid/
499000
https://www.zillow.comhttps://www.zillow.com/homedetails/7109-Cherry-Meadow-Dr-Austin-TX-78745/29494804_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/2709-Cheney-Cv-Austin-TX-78745/29504159_zpid/
375000
https://www.zillow.comhttps://www.zillow.com/homedetails/10404-Garbacz-Dr-Austin-TX-78748/58317151_zpid/
559999
https://www.zillow.comhttps://www.zillow.com/homedetails/10029-Dawn-Pearl-Dr-Austin-TX-78748/2065599638_zpid/
624999
https://www.zillow.comhttps://www.zillow.com/homedetails/10020-Dawn-Pearl-Dr-Austin-TX-78748/2065599610_zpid/
235000
https://www.zillow.comhttps://www.zillow.com/homedetails/11302-Circle-Dr-Austin-TX-78748/120899850_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/8908-Parker-Ranch-Cir-UNIT-A-Austin-TX-78748/241930738_zpid/
535000
https://www.zillow.comhttps://www.zillow.com/homedetails/6811-Cannonleague-Dr-Austin-TX-78745/29493420_zpid/
465000
https://www.zillow.comhttps://www.zillow.com/homedetails/2701-Pectoral-Dr-Austin-TX-78748/29504106_zpid/
556990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656960_zpid/
260000
https://www.zillow.comhttps://www.zillow.com/homedetails/600-Redbud-Trl-Austin-TX-78748/125802736_zpid/
795000
https://www.zillow.comhttps://www.zillow.com/homedetails/4000-Danli-Ln-Austin-TX-78749/29499194_zpid/
600000
https://www.zillow.comhttps://www.zillow.com/homedetails/3702-Alexandria-Dr-Austin-TX-78749/29494085_zpid/
860000
https://www.zillow.comhttps://www.zillow.com/homedetails/5817-Gorham-Glen-Ln-Austin-TX-78739/58316756_zpid/
630000
https://www.zillow.comhttps://www.zillow.com/homedetails/8801-Texas-Sun-Dr-Austin-TX-78748/29506146_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/811-Sand-Hill-Branch-Dr-Austin-TX-78748/251031588_zpid/
584990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656965_zpid/
604990
https://www.zillow.comhttps://www.zillow.com/community/messinger-village/2063050147_zpid/     
482990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656959_zpid/