使用 Beautiful Soup 查找嵌套的 div/class

Question

我在看這個 URL： https://www.zillow.com/homedetails/2616-Tinmouth-St-Austin-TX-78748/64743359_zpid/

我的目標是把該死的熱情拿出來放在一個列表中。

它所在的 class 是： class="Text-c11n-8-65-2__sc-aiai24-0 eUxMDw" 。

我嘗試在 html 中走得更高：

for div in soup.findAll('div', attrs={'id':'.Home-value '}):
    print (div.text)

還嘗試了 select：

items = [item.text.strip() for item in soup.select('.ds-home-values.' )]
items

我也嘗試提取：

for t in soup.select("pfs-upsell"):
    t.extract()

每次我得到一個空列表。 我在想這種類型的 HTML 可能有一些我不明白的地方。

Answer 1

該網站是動態的。 您的元素選擇什么也沒選擇，但這也不是主要原因。 獲取數據的主要原因是數據在 html DOM 下的 json 格式並帶有注釋。

import requests
import re
import json
url='https://www.zillow.com/homes/for_sale/2_p/?searchQueryState=%7B%22pagination%22%3A%7B%22currentPage%22%3A2%7D%2C%22mapBounds%22%3A%7B%22west%22%3A-97.88981437683105%2C%22east%22%3A-97.78407096862793%2C%22south%22%3A30.136145838104586%2C%22north%22%3A30.208863801102932%7D%2C%22isMapVisible%22%3Atrue%2C%22filterState%22%3A%7B%22sort%22%3A%7B%22value%22%3A%22days%22%7D%2C%22ah%22%3A%7B%22value%22%3Atrue%7D%7D%2C%22isListVisible%22%3Atrue%2C%22mapZoom%22%3A13%7D'
r = requests.get(url,headers = {'User-Agent':'Mozilla/5.0'})

data = json.loads(re.search(r'!--(\{"queryState".*?)-->', r.text).group(1))

for item in data['cat1']['searchResults']['listResults']:
    price=item['unformattedPrice']
    print(price)
    listing_url='https://www.zillow.com'+item['detailUrl']
    print(listing_url)

Output：

470000
https://www.zillow.comhttps://www.zillow.com/homedetails/7337-Menchaca-Rd-36-Austin-TX-78745/119617777_zpid/
539500
https://www.zillow.comhttps://www.zillow.com/homedetails/3214-Barnsley-Dr-Austin-TX-78745/29488852_zpid/
530000
https://www.zillow.comhttps://www.zillow.com/homedetails/10041-Aly-May-Dr-Austin-TX-78748/119624978_zpid/
659000
https://www.zillow.comhttps://www.zillow.com/homedetails/9831-Briar-Ridge-Dr-Austin-TX-78748/29509925_zpid/
499000
https://www.zillow.comhttps://www.zillow.com/homedetails/10209-Brantley-Bnd-Austin-TX-78748/29511458_zpid/
550000
https://www.zillow.comhttps://www.zillow.com/homedetails/7509-Bender-Dr-Austin-TX-78749/29486187_zpid/
585000
https://www.zillow.comhttps://www.zillow.com/homedetails/3012-Cohoba-Dr-Austin-TX-78748/251027808_zpid/
749900
https://www.zillow.comhttps://www.zillow.com/homedetails/8302-Hornet-Dr-Austin-TX-78749/29494207_zpid/
750000
https://www.zillow.comhttps://www.zillow.com/homedetails/7400-Trede-Cv-Austin-TX-78745/29497905_zpid/
630000
https://www.zillow.comhttps://www.zillow.com/homedetails/8610-Coastal-Dr-Austin-TX-78749/29499263_zpid/
815000
https://www.zillow.comhttps://www.zillow.com/homedetails/10800-Harley-Ave-Austin-TX-78748/70332585_zpid/
559900
https://www.zillow.comhttps://www.zillow.com/homedetails/8317-Empress-Blvd-Austin-TX-78745/63440918_zpid/
642000
https://www.zillow.comhttps://www.zillow.com/homedetails/8901-Curlew-Dr-Austin-TX-78748/29504016_zpid/
599000
https://www.zillow.comhttps://www.zillow.com/homedetails/405-Hogg-Pecan-Pass-Austin-TX-78748/80097626_zpid/
479000
https://www.zillow.comhttps://www.zillow.com/homedetails/2505-Jesse-Owens-Dr-Austin-TX-78748/70335498_zpid/
649000
https://www.zillow.comhttps://www.zillow.com/homedetails/2300-Yvette-Cv-Austin-TX-78748/119623072_zpid/
600000
https://www.zillow.comhttps://www.zillow.com/homedetails/8104-Cattle-Dr-Austin-TX-78749/29495648_zpid/
995000
https://www.zillow.comhttps://www.zillow.com/homedetails/3423-Buck-Race-St-Austin-TX-78748/29511556_zpid/
550000
https://www.zillow.comhttps://www.zillow.com/homedetails/7337-Menchaca-Rd-16-Austin-TX-78745/119632513_zpid/
699000
https://www.zillow.comhttps://www.zillow.com/homedetails/2601-Burly-Oak-Dr-Austin-TX-78745/29498040_zpid/
479000
https://www.zillow.comhttps://www.zillow.com/homedetails/2606-Lindsey-Cv-Austin-TX-78748/29506986_zpid/
499000
https://www.zillow.comhttps://www.zillow.com/homedetails/7109-Cherry-Meadow-Dr-Austin-TX-78745/29494804_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/2709-Cheney-Cv-Austin-TX-78745/29504159_zpid/
375000
https://www.zillow.comhttps://www.zillow.com/homedetails/10404-Garbacz-Dr-Austin-TX-78748/58317151_zpid/
559999
https://www.zillow.comhttps://www.zillow.com/homedetails/10029-Dawn-Pearl-Dr-Austin-TX-78748/2065599638_zpid/
624999
https://www.zillow.comhttps://www.zillow.com/homedetails/10020-Dawn-Pearl-Dr-Austin-TX-78748/2065599610_zpid/
235000
https://www.zillow.comhttps://www.zillow.com/homedetails/11302-Circle-Dr-Austin-TX-78748/120899850_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/8908-Parker-Ranch-Cir-UNIT-A-Austin-TX-78748/241930738_zpid/
535000
https://www.zillow.comhttps://www.zillow.com/homedetails/6811-Cannonleague-Dr-Austin-TX-78745/29493420_zpid/
465000
https://www.zillow.comhttps://www.zillow.com/homedetails/2701-Pectoral-Dr-Austin-TX-78748/29504106_zpid/
556990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656960_zpid/
260000
https://www.zillow.comhttps://www.zillow.com/homedetails/600-Redbud-Trl-Austin-TX-78748/125802736_zpid/
795000
https://www.zillow.comhttps://www.zillow.com/homedetails/4000-Danli-Ln-Austin-TX-78749/29499194_zpid/
600000
https://www.zillow.comhttps://www.zillow.com/homedetails/3702-Alexandria-Dr-Austin-TX-78749/29494085_zpid/
860000
https://www.zillow.comhttps://www.zillow.com/homedetails/5817-Gorham-Glen-Ln-Austin-TX-78739/58316756_zpid/
630000
https://www.zillow.comhttps://www.zillow.com/homedetails/8801-Texas-Sun-Dr-Austin-TX-78748/29506146_zpid/
450000
https://www.zillow.comhttps://www.zillow.com/homedetails/811-Sand-Hill-Branch-Dr-Austin-TX-78748/251031588_zpid/
584990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656965_zpid/
604990
https://www.zillow.comhttps://www.zillow.com/community/messinger-village/2063050147_zpid/     
482990
https://www.zillow.comhttps://www.zillow.com/community/copperstone/2069656959_zpid/