具有相同 class 的多个 span 标签得到特定的 webscraping with beautiful soup

Question

我无法从具有相同 class 的 span 标签获取数值

这就是 html 的样子

<ul class="sds-definition-list review-breakdown--list">
<li>
<span class="sds-definition-list__display-name">Comfort</span>
<span class="sds-definition-list__value">5.0</span>
</li>
<li>
<span class="sds-definition-list__display-name">Interior design</span>
<span class="sds-definition-list__value">4.0</span>
</li>
<li>
<span class="sds-definition-list__display-name">Performance</span>
<span class="sds-definition-list__value">5.0</span>
</li>
<li>
<span class="sds-definition-list__display-name">Value for the money</span>
<span class="sds-definition-list__value">5.0</span>
</li>
<li>
<span class="sds-definition-list__display-name">Exterior styling</span>
<span class="sds-definition-list__value">5.0</span>
</li>
<li>
<span class="sds-definition-list__display-name">Reliability</span>
<span class="sds-definition-list__value">5.0</span>
</li>
</ul>

我基本上想获取所有数值并将它们放在不同的列中，这就是我在代码中使用的内容

ua = UserAgent()
header = {'User-Agent':str(ua.safari)}
url = 'https://www.cars.com/research/nissan-leaf-2011/consumer-reviews/?page=1'
response = requests.get(url, headers=header)
print(response)
html_soup = BeautifulSoup(response.text, 'lxml')
content_list = html_soup.find_all('div', attrs={'class': 'consumer-review-container'})
data = []

for e in content_list:
    data.append({
      'review_title': e.h3.text,
      'review_content': e.select_one('p.review-body').text,
      'overall_rating': e.select_one('span.sds-rating__count').text,
      'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
      'review_date':e.find("div", {"class":"review-byline"}).div.text,
    })

对于列表数据，我想添加以下信息：舒适度、内饰、性能、物有所值、外观造型和可靠性，我想从前面提到的 html 代码中获取这些信息。

Answer 1

我相信你正在寻找这样的东西。 添加显示名称和评级列表。

data = []

for e in content_list:
    data.append({
      'review_title': e.h3.text,
      'review_content': e.select_one('p.review-body').text,
      'overall_rating': e.select_one('span.sds-rating__count').text,
      'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
      'review_date':e.find("div", {"class":"review-byline"}).div.text,
      'rating breakdown display-name' : [d.text for d in e.select('.sds-definition-list__display-name')],
      'rating breakdown value' : [d.text for d in e.select('.sds-definition-list__value')]
    })

print(data)

Output :

[{'review_title': 'Great Electric Car!', 'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!', 'overall_rating': '4.7', 'reviewer_name': 'By EVs are the future from Tucson, AZ', 'review_date': 'February 24, 2020', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '3.0', '5.0']}, {'review_title': 'Fun to drive, and no gasoline!!', 'review_content': 'For an around-the-town running errands car, this car is fabulous.  Fun to drive, and super-low cost per mile (rated at over 99 mpg compared to regular gas cars).', 'overall_rating': '4.7', 'reviewer_name': 'By EV fan from Tucson, AZ', 'review_date': 'December 23, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '5.0', '5.0', '5.0', '5.0']}, {'review_title': 'Save movey Save Environment', 'review_content': 'It’s a a wonderful buy and I enjoy it immensely especially the fact that I don’t have to get oil changes brake changes fluids etc\nIt’s a perfect buy', 'overall_rating': '5.0', 'reviewer_name': 'By BayArea from CA', 'review_date': 'November  2, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '5.0', '5.0']}, {'review_title': 'Good money saver', 'review_content': "Car has saved me a lot of money.  Electricity is much cheaper than gas, and you can't buy a car at this price, age, and mileage.  \n\nThe one downside to the Leaf is the battery replacement cost is very high.  Nissan raised the battery replacement cost after we bought the car, which was very frustrating.", 'overall_rating': '4.9', 'reviewer_name': 'By Lothsahn from Missouri', 'review_date': 'July  8, 2019', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '5.0', '4.0']}, {'review_title': 'What I expected', 'review_content': "There was nothing new to this leaf. I've researched them a lot and test drove one other one. This one was on par with what I've seen/drove. My mine focus was on the drive range depletion while driving. I drove 20 miles but only lost 15 miles in drive range thanks to the braking regeneration.", 'overall_rating': '3.8', 'reviewer_name': 'By brad', 'review_date': 'May 31, 2018', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '4.0', '4.0', '3.0', '—']}, {'review_title': "It's like driving the future!", 'review_content': "This car has been a great commuter car.  It's been reliable and easy to maintain!  The electric torque and power makes such a fun driving experience! Charging this car with a full charge only costs about $1, to provide up to about 70-80 miles commute.  The A/C is by far the best performing of any car I have owned!", 'overall_rating': '4.4', 'reviewer_name': 'By evcarguy from Chandler, AZ', 'review_date': 'November 23, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '4.0', '5.0', '4.0', '5.0']}, {'review_title': 'Nice car with no cost for fuel', 'review_content': 'Nissan leaf is a nice car without any additional cost for fuel as well as maintenance for daily communication...', 'overall_rating': '4.0', 'reviewer_name': 'By Zarni Tun from Rowland Heights,CA', 'review_date': 'July 28, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '3.0', '4.0', '5.0', '3.0', '3.0']}, {'review_title': 'No gas stations. No maintenance.', 'review_content': "I don't drive very much.  My LEAF has been worry free for 6 years.  NOTHING has gone wrong with it.  I expect it will serve me for 20 years.  I have solar panels on my roof so I drive this car for virtually $0.", 'overall_rating': '4.6', 'reviewer_name': 'By AQL from Mercer Island, WA', 'review_date': 'June 12, 2017', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['4.0', '4.0', '5.0', '5.0', '4.0', '5.0']}, {'review_title': '5 months review', 'review_content': "I got this car with 3800 miles certified pre own , 5 months ago , now at 15000 miles the cars its saving me money on gas i drive around 80-95 miles a day with no problem , you have to be careful not to be speeding around town but order that that my car will give an average of 90 miles on a charge, i drive around 2000 miles a month so the money i am saving on gas is a lo t, pretty much the car pay it self , the more you drive the more you save, i haven't lost any bars yet , i know that's coming , i will continue to use and abuse my car , until that day comes around.", 'overall_rating': '4.9', 'reviewer_name': 'By JT from kissimmee, fl', 'review_date': 'November 26, 2014', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '5.0', '5.0', '4.0', '5.0']}, {'review_title': 'Problems with battery capacity long-term', 'review_content': 'We have owned our Leaf since May 2011. We have loved the car but are now getting quite concerned. My husband drives the car, on average, 20-40 miles/day to and from work and running errands, mostly 100% on city roads. We live in San Diego, so no issue with winter weather and we live 7 miles from the ocean so seldom have daytime temperatures above 85. Originally, we would get 65-70 miles per 80-90% charge. Last fall we noticed that there was considerably less remaining charge left after a day of driving. He began to track daily miles, remaining "bars", as well as started charging it 100%. For 9 months we have only been getting 40-45 miles on a full charge with only 1-2 "bars" remaining at the end of the day. Sometimes it will be blinking and "talking" to us to get to a charging place ASAP. We just had it into the dealership. Though on a full charge, the car gauge shows 12 bars, the dealership states that the batteries have lost 2 bars via the computer diagnostics (which we are told is a different reading from the car gauge itself) and, that they say, is average and excepted for the car at this age. Everything else (software, diagnostics, etc.) shows 100%, so the dealership thinks that the car is functioning as it should. They are unable to explain why we can only go 40-45 miles on a charge, but keep saying that the car tests out fine. If the distance one is able to drive on a full charge decreases any further, it will begin to render the car useless. As someone else recommended, in retrospect, the best way to go is to lease the Leaf so that battery life is not an issue.', 'overall_rating': '3.7', 'reviewer_name': 'By Cathie from San Diego', 'review_date': 'July 21, 2014', 'rating breakdown display-name': ['Comfort', 'Interior design', 'Performance', 'Value for the money', 'Exterior styling', 'Reliability'], 'rating breakdown value': ['5.0', '5.0', '2.0', '3.0', '5.0', '3.0']}]

Answer 2

根据您的评论，它正在工作。每个li tag都有两个子 span 标签和span:-soup-contains()包括文本值将 select 来自 li 的第一个 span 和~将捕获紧邻的下一个标签，即 li 的第二个span标签

from bs4 import BeautifulSoup
import requests

#ua = UserAgent()
header = {'User-Agent':'mozila/5.0'}
url = 'https://www.cars.com/research/nissan-leaf-2011/consumer-reviews/?page=1'
response = requests.get(url, headers=header)
print(response)
html_soup = BeautifulSoup(response.text, 'lxml')
content_list = html_soup.find_all('div', attrs={'class': 'consumer-review-container'})
data = []

for e in content_list:
    data.append({
      'review_title': e.h3.text,
      'review_content': e.select_one('p.review-body').text,
      'overall_rating': e.select_one('span.sds-rating__count').text,
      'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
      'review_date':e.find("div", {"class":"review-byline"}).div.text,
      'comfort':e.select_one('span:-soup-contains("Comfort")~span').text,
      'Interior design':e.select_one('span:-soup-contains("Interior design")~span').text,
      'Performance':e.select_one('span:-soup-contains("Performance")~span').text,
      'Value for the money':e.select_one('span:-soup-contains("Value for the money")~span').text,
      'Exterior styling':e.select_one('span:-soup-contains("Exterior styling")~span').text,
      'Reliability':e.select_one('span:-soup-contains("Reliability")~span').text
    })

print(data)

Output：

<Response [200]>
[{'review_title': 'Great Electric Car!', 'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly 
and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!', 'overall_rating': '4.7', 'reviewer_name': 'By EVs are the future from Tucson, AZ', 'review_date': 'February 24, 2020', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '3.0', 'Reliability': '5.0'}, 
{'review_title': 'Fun to drive, and no gasoline!!', 'review_content': 'For an around-the-town 
running errands car, this car is fabulous.  Fun to drive, and super-low cost per mile (rated at over 99 mpg compared to regular gas cars).', 'overall_rating': '4.7', 'reviewer_name': 'By EV fan from Tucson, AZ', 'review_date': 'December 23, 2019', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '5.0'}, {'review_title': 'Save movey Save Environment', 'review_content': 'It’s a a 
wonderful buy and I enjoy it immensely especially the fact that I don’t have to get oil changes brake changes fluids etc\nIt’s a perfect buy', 'overall_rating': '5.0', 'reviewer_name': 'By BayArea from CA', 'review_date': 'November  2, 2019', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '5.0'}, {'review_title': 'Good money saver', 'review_content': "Car has saved me a lot of money.  Electricity is much cheaper than gas, and you can't buy a car at this price, age, and mileage.  \n\nThe one downside to the Leaf is the battery replacement cost is very high.  Nissan raised the battery replacement cost after we bought the car, which was very frustrating.", 'overall_rating': '4.9', 'reviewer_name': 'By Lothsahn from Missouri', 'review_date': 'July  
8, 2019', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '5.0', 'Reliability': '4.0'}, {'review_title': 'What I expected', 'review_content': "There was nothing new to this leaf. I've researched them a lot and test drove one other one. This one was on par with what I've seen/drove. My mine focus was on the 
drive range depletion while driving. I drove 20 miles but only lost 15 miles in drive range thanks to the braking regeneration.", 'overall_rating': '3.8', 'reviewer_name': 'By brad', 'review_date': 'May 31, 2018', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '4.0', 'Value for the money': '4.0', 'Exterior styling': '3.0', 'Reliability': '—'}, {'review_title': "It's like driving the future!", 'review_content': "This car has been a great commuter car.  It's been reliable and easy to maintain!  The electric torque and power makes such a fun driving experience! Charging this car with a full charge only costs about $1, to provide up to about 
70-80 miles commute.  The A/C is by far the best performing of any car I have owned!", 'overall_rating': '4.4', 'reviewer_name': 'By evcarguy from Chandler, AZ', 'review_date': 'November 23, 2017', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '4.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': 'Nice car with 
no cost for fuel', 'review_content': 'Nissan leaf is a nice car without any additional cost for fuel as well as maintenance for daily communication...', 'overall_rating': '4.0', 'reviewer_name': 'By Zarni Tun from Rowland Heights,CA', 'review_date': 'July 28, 2017', 'comfort': '5.0', 'Interior design': '3.0', 'Performance': '4.0', 'Value for the money': '5.0', 'Exterior styling': '3.0', 'Reliability': '3.0'}, {'review_title': 'No gas stations. No maintenance.', 'review_content': "I don't drive very much.  My LEAF has been worry free for 6 years.  NOTHING has gone wrong with it.  I expect it will serve me for 20 years.  I have solar panels on my roof 
so I drive this car for virtually $0.", 'overall_rating': '4.6', 'reviewer_name': 'By AQL from Mercer Island, WA', 'review_date': 'June 12, 2017', 'comfort': '4.0', 'Interior design': '4.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': '5 months review', 'review_content': "I got this car with 3800 miles certified pre own , 5 months ago , now at 15000 miles the cars its saving me money on gas i drive around 80-95 miles a day with no problem , you have to be careful not to be speeding around town but order that that my car will give an average of 90 miles on a charge, i drive around 2000 miles a month so the money i am saving on gas is a lo t, pretty much the car pay it self , the more you drive the more you save, i haven't lost any bars yet , i know that's coming , i will continue to use and abuse my car , until that day comes around.", 'overall_rating': 
'4.9', 'reviewer_name': 'By JT from kissimmee, fl', 'review_date': 'November 26, 2014', 'comfort': '5.0', 'Interior design': '5.0', 'Performance': '5.0', 'Value for the money': '5.0', 'Exterior styling': '4.0', 'Reliability': '5.0'}, {'review_title': 'Problems with battery capacity long-term', 'review_content': 'We have owned our Leaf since May 2011. We have loved the car but are now getting quite concerned. My husband drives the car, on average, 20-40 miles/day to 
and from work and running errands, mostly 100% on city roads. We live in San Diego, so no issue with winter weather and we live 7 miles from the ocean so seldom have daytime temperatures above 85. Originally, we would get 65-70 miles per 80-90% charge. Last fall we noticed that there was considerably less remaining charge left after a day of driving. He began to track daily miles, remaining "bars", as well as started charging it 100%. For 9 months we have only been 
getting 40-45 miles on a full charge with only 1-2 "bars" remaining at the end of the day. Sometimes it will be blinking and "talking" to us to get to a charging place ASAP. We just had it into the dealership. Though on a full charge, the car gauge shows 12 bars, the dealership states that the batteries have lost 2 bars via the computer diagnostics (which we are told is a different reading from the car gauge itself) and, that they say, is average and excepted for the car at this age. Everything else (software, diagnostics, etc.) shows 100%, so the dealershipcar tests out fine. If the distance one is able to drive on a full chargeain why we can only g decreases any further, it will begin to render the car useless. As someof the distance one isne else recommended, in retrospect, the best way to go is to lease the Leender the car uselessaf so that battery life is not an issue.', 'overall_rating': '3.7', 'reviase the Leaf so that ewer_name': 'By Cathie from San Diego', 'review_date': 'July 21, 2014', ' 'By Cathie from San comfort': '5.0', 'Interior design': '5.0', 'Performance': '2.0', 'Value fn': '5.0', 'Performanor the money': '3.0', 'Exterior styling': '5.0', 'Reliability': '3.0'}] 

Answer 3

要获得结果，您可以遍历<li>并在dict comprehension中使用.stripped_strings提取内容，然后将现有dict和 append 更新为data 。

创建一个DataFrame这将为每个项目创建单独的列：

for e in content_list:
    d = {
      'review_title': e.h3.text,
      'review_content': e.select_one('p.review-body').text,
      'overall_rating': e.select_one('span.sds-rating__count').text,
      'reviewer_name':e.select_one("div.review-byline div:nth-of-type(2)").text,
      'review_date':e.find("div", {"class":"review-byline"}).div.text,
    }

    d.update(dict(s.stripped_strings for s in e.select('ul.sds-definition-list li')))

    data.append(d)
data

Output：

[{'review_title': 'Great Electric Car!',
  'review_content': 'This is the perfect electric car for driving around town, doing errands or even for a short daily commuter. It is very comfy and very quick. The only issue was the first gen battery. The 2011-2014 battery degraded quickly and if the owner did not have Nissan replace it, all those cars are now junk and can only go 20 miles or so on a charge. We had Nissan replace our battery with the 2nd gen battery and it is good as new!',
  'overall_rating': '4.7',
  'reviewer_name': 'By EVs are the future from Tucson, AZ',
  'review_date': 'February 24, 2020',
  'Comfort': '5.0',
  'Interior design': '5.0',
  'Performance': '5.0',
  'Value for the money': '5.0',
  'Exterior styling': '3.0',
  'Reliability': '5.0'},...]