[英]How to find nested html elements using lxml in Python?
我正在尝试抓取以下 html。
有多个 div,其中 class="review-card"。
这些 div 中的每一个始终包含一个脚本元素,其中 data-initial-state="data-always-exist",有时还包含一个脚本元素,其中 data-initial-state="data-may-not-exist"。
我想从这两个脚本元素中检索数据。 当第二个不存在时,我想返回一个特定的值,例如 0。
正如您在下面的代码中看到的那样,我设法找到了“retrieve-card”div 元素。 但是,我无法检索每个 div 元素内的脚本元素。 我的代码总是返回一个列表而不是单个元素。 我究竟做错了什么?
<html>
<body>
<main>
<div class="review-list">
<div class="review-card">
<article class="review">
<script type="application.json" data-initial-state="data-always-exist">
{"reviewBody":"Brilliant value","stars":5}
</script>
<section class="review__content">
<div class="content">
<script type="application.json" data-initial-state="data-may-not-exist">
{"isVerified":true,"verificationSource":"invitation"}
</script>
</div>
</section>
</article>
</div>
<div class="review-card">
<article class="review">
<script type="application.json data-initial-state="data-always-exist">
{"reviewBody":"Brilliant value","stars":5}
</script>
</article>
</div>
<div class="review-card">
<article class="review">
<script type="application.json" data-initial-state="data-always-exist">
{"reviewBody":"Great","stars":4}
</script>
<section class="review__content">
<div class="content">
<script type="application.json" data-initial-state="data-may-not-exist">
{"isVerified":false,"verificationSource":"invitation"}
</script>
</div>
</section>
</article>
</div>
</div>
</main>
</body>
</html>
我尝试了以下方法:
from lxml import html
import requests
page = requests.get('http://somewebsite.com')
tree = html.fromstring(page.content)
#finds the review list
review_list = tree.xpath('//div[@class="review-list"]')
#finds all the review cards
review_cards = review_list[0].xpath('//div[contains(@class,"review-card")]')
for card in review_cards:
#this part of the code does not work as intended -returns a list vs a single items.
data_always_exist = card.xpath("//script[starts-with(@data-initial-state, 'data-always-exist')]")
data_not_always_exist = card.xpath("//script[starts-with(@data-initial-state, 'data-may-not-exist')]")
使用beautifulsoup
的解决方案:
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get("http://somewebsite.com").content, "lxml")
for card in soup.select(".review-card"):
print("data-always-exist:")
d = card.select_one('[data-initial-state="data-always-exist"]')
if d:
print(d.contents[0].strip())
print("data-may-not-exist:")
d = card.select_one('[data-initial-state="data-may-not-exist"]')
if d:
print(d.contents[0].strip())
print("-" * 80)
印刷:
data-always-exist:
{"reviewBody":"Brilliant value","stars":5}
data-may-not-exist:
{"isVerified":true,"verificationSource":"invitation"}
--------------------------------------------------------------------------------
data-always-exist:
{"reviewBody":"Brilliant value","stars":5}
data-may-not-exist:
--------------------------------------------------------------------------------
data-always-exist:
{"reviewBody":"Great","stars":4}
data-may-not-exist:
{"isVerified":false,"verificationSource":"invitation"}
--------------------------------------------------------------------------------
带有lxml
的版本(在 XPath 中使用点 ( .
)):
# ...
tree = html.fromstring(page.content)
cards = tree.xpath('//div[contains(@class,"review-card")]')
for card in cards:
# this part of the code does not work as intended -returns a list vs a single items.
data_always_exist = card.xpath(
".//script[starts-with(@data-initial-state, 'data-always-exist')]"
)
data_not_always_exist = card.xpath(
".//script[starts-with(@data-initial-state, 'data-may-not-exist')]"
)
print(data_always_exist)
print(data_not_always_exist)
print("-" * 80)
印刷:
[<Element script at 0x7fc202aadd10>]
[<Element script at 0x7fc202aade50>]
--------------------------------------------------------------------------------
[<Element script at 0x7fc202aadea0>]
[]
--------------------------------------------------------------------------------
[<Element script at 0x7fc202aade50>]
[<Element script at 0x7fc202aadea0>]
--------------------------------------------------------------------------------
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.