[英]How to find nested html elements using lxml in Python?
我正在嘗試抓取以下 html。
有多個 div,其中 class="review-card"。
這些 div 中的每一個始終包含一個腳本元素,其中 data-initial-state="data-always-exist",有時還包含一個腳本元素,其中 data-initial-state="data-may-not-exist"。
我想從這兩個腳本元素中檢索數據。 當第二個不存在時,我想返回一個特定的值,例如 0。
正如您在下面的代碼中看到的那樣,我設法找到了“retrieve-card”div 元素。 但是,我無法檢索每個 div 元素內的腳本元素。 我的代碼總是返回一個列表而不是單個元素。 我究竟做錯了什么?
<html>
<body>
<main>
<div class="review-list">
<div class="review-card">
<article class="review">
<script type="application.json" data-initial-state="data-always-exist">
{"reviewBody":"Brilliant value","stars":5}
</script>
<section class="review__content">
<div class="content">
<script type="application.json" data-initial-state="data-may-not-exist">
{"isVerified":true,"verificationSource":"invitation"}
</script>
</div>
</section>
</article>
</div>
<div class="review-card">
<article class="review">
<script type="application.json data-initial-state="data-always-exist">
{"reviewBody":"Brilliant value","stars":5}
</script>
</article>
</div>
<div class="review-card">
<article class="review">
<script type="application.json" data-initial-state="data-always-exist">
{"reviewBody":"Great","stars":4}
</script>
<section class="review__content">
<div class="content">
<script type="application.json" data-initial-state="data-may-not-exist">
{"isVerified":false,"verificationSource":"invitation"}
</script>
</div>
</section>
</article>
</div>
</div>
</main>
</body>
</html>
我嘗試了以下方法:
from lxml import html
import requests
page = requests.get('http://somewebsite.com')
tree = html.fromstring(page.content)
#finds the review list
review_list = tree.xpath('//div[@class="review-list"]')
#finds all the review cards
review_cards = review_list[0].xpath('//div[contains(@class,"review-card")]')
for card in review_cards:
#this part of the code does not work as intended -returns a list vs a single items.
data_always_exist = card.xpath("//script[starts-with(@data-initial-state, 'data-always-exist')]")
data_not_always_exist = card.xpath("//script[starts-with(@data-initial-state, 'data-may-not-exist')]")
使用beautifulsoup
的解決方案:
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get("http://somewebsite.com").content, "lxml")
for card in soup.select(".review-card"):
print("data-always-exist:")
d = card.select_one('[data-initial-state="data-always-exist"]')
if d:
print(d.contents[0].strip())
print("data-may-not-exist:")
d = card.select_one('[data-initial-state="data-may-not-exist"]')
if d:
print(d.contents[0].strip())
print("-" * 80)
印刷:
data-always-exist:
{"reviewBody":"Brilliant value","stars":5}
data-may-not-exist:
{"isVerified":true,"verificationSource":"invitation"}
--------------------------------------------------------------------------------
data-always-exist:
{"reviewBody":"Brilliant value","stars":5}
data-may-not-exist:
--------------------------------------------------------------------------------
data-always-exist:
{"reviewBody":"Great","stars":4}
data-may-not-exist:
{"isVerified":false,"verificationSource":"invitation"}
--------------------------------------------------------------------------------
帶有lxml
的版本(在 XPath 中使用點 ( .
)):
# ...
tree = html.fromstring(page.content)
cards = tree.xpath('//div[contains(@class,"review-card")]')
for card in cards:
# this part of the code does not work as intended -returns a list vs a single items.
data_always_exist = card.xpath(
".//script[starts-with(@data-initial-state, 'data-always-exist')]"
)
data_not_always_exist = card.xpath(
".//script[starts-with(@data-initial-state, 'data-may-not-exist')]"
)
print(data_always_exist)
print(data_not_always_exist)
print("-" * 80)
印刷:
[<Element script at 0x7fc202aadd10>]
[<Element script at 0x7fc202aade50>]
--------------------------------------------------------------------------------
[<Element script at 0x7fc202aadea0>]
[]
--------------------------------------------------------------------------------
[<Element script at 0x7fc202aade50>]
[<Element script at 0x7fc202aadea0>]
--------------------------------------------------------------------------------
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.