美丽的汤不返回HTML文件中的所有内容吗？

Question

HTML noob here, so I could be misunderstanding something about the HTML document, so bear with me.

I'm using Beautiful Soup to parse web data in Python. Here is my code:

import urllib
import BeautifulSoup

url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup.BeautifulSoup(page)
indicateGameDone = str(soup.find("div", {"class": "nbaModTopStatus"}))
print indicateGameDone

now, if you look at the website, the HTML code has the line <p class="nbaLiveStatTxSm"> FINAL </p> , (inspect the 'Final' text on the left side of the container on the first ATL-WAS game on the page to see it for youself.) But when I run the code above, my code doesn't return the 'FINAL' that is seen on the webpage, and instead the nbaLiveStatTxSm class is empty.

On my machine, this is the output when I print indicateGameDone :

<div class="nbaModTopStatus"><p class="nbaLiveStatTx">Live</p><p class="nbaLiveStatTxSm"></p><p class="nbaFnlStatTx">Final</p><p class="nbaFnlStatTxSm"></p></div>

Does anyone know why this is happening?

EDIT: clarification: the problem isn't retrieving the text within the tag, the problem is that when I take the html code from the website and print it out in python, something that I saw when I inspected the element on the web is not there in the print statement in Python.

Answer 1

You can use this logic to extract any text. This code allows you to extract any data between any tags. Output - FINAL

import urllib
from bs4 import BeautifulSoup
url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url)
soup = BeautifulSoup(page)
indicateGameDone = soup.find("div", {"class": "nbaFnlStatTx"})
for p in indicateGameDone:
    p_text = soup.find("p", {"class": "nbaFnlStatTxSm"})
    print(p_text.getText())
    break;

Answer 2

It looks like your problem is not with BeautifulSoup but instead with urllib.

Try running the following commands

>>> import urllib
>>> url = "http://www.nba.com/gameline/20160323/"
>>> page = urllib.urlopen(url).read()
>>> page.find('<div class="nbaModTopStatus">')
44230

Which is no surprise considering that Beautiful Soup was able to find the div itself. However when we look a little deeper into what urllib is actually collecting we can see that the <p class="nbaFnlStatTxSm"> is empty by running

>>> page[44230:45000]
'<div class="nbaModTopStatus"><p class="nbaLiveStatTx">Live</p><p class="nbaLiveStatTxSm"></p><p class="nbaFnlStatTx">Final</p><p class="nbaFnlStatTxSm"></p></div><div id="nbaGLBroadcast"><a href="/leaguepass"><img src="/.element/img/3.0/sect/gameline/broadcasters/lp.png"></a></div><div class="nbaTeamsRow"><div class="nbaModTopTeamScr nbaModTopTeamAw"><h5 class="nbaModTopTeamName awayteam">ATL</h5><img src="http://i.cdn.turner.com/nba/nba/.element/img/2.0/sect/gameline/teams/ATL.gif" width="34" height="22" title="Atlanta Hawks"><h4 class="nbaModTopTeamNum  win"></h4></div><div class="nbaModTopTeamScr nbaModTopTeamHm"><h5 class="nbaModTopTeamName hometeam">WAS</h5><img src="http://i.cdn.turner.com/nba/nba/.element/img/2.0/sect/gameline/teams/WAS.gif" width="34" '

You can see that the tag is empty, so your problem is the data that's being passed to Beautiful Soup, not the package itself.

Answer 3

1. changed the import of beautifulsoup to the proper syntax for the current version of BeautifulSoup
2. corrected the way you were constructing the BeautifulSoup object
3. fixed your find statement, then used the .text command to get the string representation of the text in the HTML you're after.

With some minor modifications to your code as listed above, your code runs for me.

import urllib
from bs4 import BeautifulSoup

url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup(page)
indicateGameDone = soup.find("div", {"class": "nbaModTopStatus"})
print indicateGameDone.text ## "LiveFinal "

to address comments:

import urllib
from bs4 import BeautifulSoup

url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup(page)
indicateGameDone = soup.find("p", {"class": "nbaFnlStatTx"})
print indicateGameDone.text