使用python BeautifulSoup进行HTML抓取

Question

I have following HTML file and I am trying to scrape the complete sentence using BeautifulSoup but couldn't get it. Currently I am getting only highlighted words. my desired output should be

Antenna booster has stopped sending signal files ,possible user network issue or BOOSTER issue.

Any solution?

  </table>
  <!--Record Header End-->
  <span style="BACKGROUND-COLOR: #0000ff; color: #ffffff">
   Antenna
  </span>
  <span style="BACKGROUND-COLOR: #0000ff; color: #ffffff">
   booster
  </span>
  has stopped
  <span style="BACKGROUND-COLOR: #0000ff; color: #ffffff">
   sending
  </span>
  signal files ,possible user
  <span style="BACKGROUND-COLOR: #0000ff; color: #ffffff">
   network
  </span>
  <span style="BACKGROUND-COLOR: #ff0000">
   issue
  </span>
  or BOOSTER
  <span style="BACKGROUND-COLOR: #ff0000">
   issue
  </span>
  .
  <br>
   <br>
    <br>

Here is what I tried:

issue_field = soup.find_all('span', {'style':'BACKGROUND-COLOR: #0000ff; color: #ffffff'}) 
issue_str = str(issue_field) 
Issue_corpora = [word.lower() for word in BeautifulSoup(issue_str,'html.parser').get_text().strip().sp‌​lit(',')]
print(Issue_corpora)

Answer 1

The problem is there are texts outside the elements. There is a duplicate question on SO already: Get text outside known element beautifulsoup

So here is the solution, maybe needs a little polishing. (note that variable t contains the html as text)

from bs4 import BeautifulSoup as bs
soup = bs(t)
text = ''
for span in soup.findAll('span'):
    text += getattr(span, 'text', '').strip() + ' '
    text += getattr(span, 'nextSibling', '').strip() + ' '

Result using this approach is:

>>> In : text
>>> Out: u'Antenna  booster has stopped sending signal files ,possible user network  issue or BOOSTER issue . '

You can replace doubled spaces with a single space or remove space before comma or define rules to handle it while looping through span elements.

Answer 2

from bs4 import BeautifulSoup
import re 

example = """your example""" 

soup = BeautifulSoup(example, "html.parser")

_text = ""
for span in soup.find_all('span', style=re.compile('BACKGROUND-COLOR:')):
    _text += "%s %s" % (span.get_text(strip=True), span.next_sibling.replace("\n", ""))

print (re.sub(" +"," ", _text))

Use re in the end to trim extra spaces.

Outputs:

Antenna booster has stopped sending signal files ,possible user network issue or BOOSTER issue .