如何在BeautifulSoup中将html提取为文本？

Question

I am using the following code to browse an html page and try to get the data i need using BeautifulSoup. everything looks good but i am hitting a wall and got stuck.

What i need to accomplish is to extract this 9h7a2m value from this line :

D: string-1.string2 15030 9h7a2m string3

The result im getting is this :

<p>D: string-1.string2 15030 9h7a2m string3.string<br/>
D: string-1.string2 15030 9h7a2m string3.string<br/>
D: string-1.string2 15030 9h7a2m string3.string</p>
<p><span id="more-1203"></span></p>
<p>D: string-1.string2 15030 9h7a2m string3.string<br/>
D: string-1.string2 15030 9h7a2m string3.string<br/>
D: string-1.string2 15030 9h7a2m string3.string<br/>
D: string-1.string2 15030 9h7a2m string3.string<br/>
<p>pinging test is positive but no works</p>
<p>how much time are online?</p>
<p><input aria-required="true" id="author" name="author" size="22" tabindex="1" type="text" value=""/>
<label for="author"><small>Name (required)</small></label></p>
<p><input aria-required="true" id="email" name="email" size="22" tabindex="2" type="text" value=""/>
<label for="email"><small>Mail (will not be published) (required)</small></label></p>
<p><input id="url" name="url" size="22" tabindex="3" type="text" value=""/>
<label for="url"><small>Website</small></label></p>
<p><textarea cols="100%" id="comment" name="comment" rows="10" tabindex="4"></textarea></p>
<p><input id="submit" name="submit" tabindex="5" type="submit" value="Submit Comment"/>
<input id="comment_post_ID" name="comment_post_ID" type="hidden" value="41"/>
<input id="comment_parent" name="comment_parent" type="hidden" value="0"/>
</p>
<p style="display: none;"><input id="akismet_comment_nonce" name="akismet_comment_nonce" type="hidden" value="1709964457"/></p>
<p style="display: none;"><input id="ak_js" name="ak_js" type="hidden" value="99"/></p>

At the end i need to save it to a text file.

My code :

import mechanize
from bs4 import BeautifulSoup



# mechanize 

mech = mechanize.Browser()
mech.set_handle_robots(True)   
mech.set_handle_refresh(True)  
mech.addheaders =  [('User-agent', 'Firefox')]
url = ('http://example.com/')
response = mech.open(url)
resp = response.read()

# beautifulsoup


soup = BeautifulSoup(resp)
soup.prettify()



# test code



for i in soup.find('div',{'id':'content'}).findAll('p'):

    print i

Thanks in advance.

Answer 1

You can extract it using a regular expression:

import re
from bs4 import BeautifulSoup

data = """your html here"""

soup = BeautifulSoup(data)

s = soup.find('p').br.previous_sibling  # find "p" element and get the part before the 1st br
match = re.search('string\-1\.string2 \d+ (\w+) string3\.string', s)
print match.group(1)

prints 9h7a2m .

---

UPD (real web-site):

from urllib2 import urlopen
from bs4 import BeautifulSoup

data = urlopen('your URL here')
soup = BeautifulSoup(data)

entry = soup.find('div', class_="entry")

for p in entry.find_all('p'):
    for row in p.find_all(text=True):
        try:
            print row.split(' ')[-2]
        except IndexError:
            continue