BeautifulSoup：如何获取带有注释的课程的非注释内容？

Question

I'm trying to get data from a webpage using BeautifulSoup . It works fine for the most data, but one class seems to work different and I can't figure out what to to. Are comments maybe affecting soup.find_all ?

So I have a webpage with several classes having the same name and I'm finding the contents with soup.find_all . While this works for the class "points column" , which always looks like this:

<div class="points column">Punkte</div>
<div class="points column">45.677</div>
<div class="points column">43.445</div>
...

It doesn't work for the class "teamValue column" , which looks like this:

<div class="teamValue column">Teamwert</div>
<div class="teamValue column">
<!-- react-text: 690 -->
554,4
<!-- /react-text -->
<!-- react-text: 691 -->
 €
<!-- /react-text -->
</div>
<div class="teamValue column">
<!-- react-text: 705 -->
449,7
<!-- /react-text -->
<!-- react-text: 706 -->
 €
<!-- /react-text -->
</div>
...

This is my code:

def getplayerdata(self):
    bot = self.bot
    soup = BeautifulSoup(bot.page_source, 'html.parser')

    playervalue = soup.find_all("div",class_="teamValue column",text=True)
    playerpoints = soup.find_all("div",class_="points column",text=True)

    print(playervalue)
    print(playerpoints)

The output for playerpoints works as expected, I get all the data and can extract only the text with the .string command.

But for playervalue I only get one element in my list, which is:

[<div class="teamValue column">Teamwert</div>]

Answer 1

I can get this text if I use find_all() without text=True and .get_text() or .text instead of .string

from bs4 import BeautifulSoup as BS

text = '''<div class="teamValue column">Teamwert</div>
<div class="teamValue column">
<!-- react-text: 690 -->
554,4
<!-- /react-text -->
<!-- react-text: 691 -->
 €
<!-- /react-text -->
</div>
<div class="teamValue column">
<!-- react-text: 705 -->
449,7
<!-- /react-text -->
<!-- react-text: 706 -->
 €
<!-- /react-text -->
</div>'''

soup = BS(text, 'html.parser')

all_items = soup.find_all('div',class_="teamValue column") #text=True)


for item in all_items:
    print('1>', item.text)

for item in all_items:
    print('2>', item.get_text(strip=True, separator=' '))

for item in all_items:
    print('3>', item.string)

Result:

1> Teamwert
1> 

554,4


 €


1> 

449,7


 €


2> Teamwert
2> 554,4 €
2> 449,7 €
3> Teamwert
3> None
3> None

Answer 2

Just change text= False :)

playervalue = soup.find_all("div",class_="teamValue column",text=False)
print(len(playervalue))

Out:

3

Answer 3

You could use soup.select and re.sub to get rid of the new lines

from bs4 import BeautifulSoup
import re

html = '''
<div class="teamValue column">Teamwert</div>
<div class="teamValue column">
<!-- react-text: 690 -->
554,4
<!-- /react-text -->
<!-- react-text: 691 -->
 €
<!-- /react-text -->
</div>
<div class="teamValue column">
<!-- react-text: 705 -->
449,7
<!-- /react-text -->
<!-- react-text: 706 -->
 €
<!-- /react-text -->
</div>'''

soup = bs(html, 'lxml')
team_values = [re.sub('\n+', '',item.text) for item in soup.select('.teamValue.column')]
print(team_values)