使用 Bs4 抓取 Python 3

Question

我正在嘗試使用 python 3（使用 PyCharm）抓取本網站的 HTML： https ://www.idealista.com/venta-viviendas/madrid-madrid/。 我只對房子的價格感興趣，所以我將搜索范圍縮小到特定范圍，如下所示：

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')

soup = BeautifulSoup(page.text, 'html.parser')


prices=soup.findAll("span", {"class": "item-price h2-simulated"})

print(len(prices))
print(prices)

當我運行它時，我得到這個：0 []

這意味着它沒有找到任何東西。 此外，如果我打印所有內容：print(soup) 對於這么大的頁面，我得到的 html 內容很少，因此顯然無法獲取所有內容。

Answer 1

該站點是動態的，因此，您需要使用瀏覽器操作工具，例如selenium ：

from bs4 import BeautifulSoup as soup
from selenium import webdriver
import re, collections, itertools
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')
homes = soup(d.page_source, 'html.parser').find_all('div', {'class':'item-info-container'})
results = [i.find('div', {'class':re.compile('price\-row')}).text for i in homes]
price = collections.namedtuple('price', ['original', 'current', 'drop'])
new_prices = [list(filter(None, re.split('\s{2,}', i))) for i in results]
final_prices = [price(a, *[a for a, _ in itertools.zip_longest(b, [None, None])]) for a, *b in new_prices]

輸出：

[price(original=' 369.000€', current='395.000 €', drop='7%'), price(original=' 1.250.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido', current='1.650.000 €', drop='4%'), price(original=' 1.750.000€ Garaje incluido', current='1.875.000 €', drop='7%'), price(original=' 1.090.000€ Garaje incluido', current='1.195.000 €', drop='9%'), price(original=' 795.000€ ', current=None, drop=None), price(original=' 1.095.000€ Garaje incluido ', current=None, drop=None), price(original=' 355.000€ Garaje incluido ', current=None, drop=None), price(original=' 995.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.130.000€ Garaje incluido', current='1.190.000 €', drop='5%'), price(original=' 850.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.200.000€ Garaje incluido ', current=None, drop=None), price(original=' 990.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.100.000€ ', current=None, drop=None), price(original=' 830.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.390.000€ Garaje incluido ', current=None, drop=None), price(original=' 685.000€ ', current=None, drop=None), price(original=' 1.150.000€ Garaje incluido', current='1.200.000 €', drop='4%'), price(original=' 915.000€ ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido ', current=None, drop=None), price(original=' 625.000€ Garaje incluido', current='640.000 €', drop='2%'), price(original=' 735.000€', current='760.000 €', drop='3%'), price(original=' 890.000€ Garaje incluido', current='950.000 €', drop='6%'), price(original=' 925.000€', current='999.000 €', drop='7%'), price(original=' 975.000€', current='1.100.000 €', drop='11%'), price(original=' 850.000€ Garaje incluido', current='870.000 €', drop='2%'), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.500.000€ ', current=None, drop=None), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.359.000€ ', current=None, drop=None)]