如何使用html2text / BeautifulSoup python删除[font]标签

Question

I'm using BeautifulSoup and get the result from my website, it's a chunk of code with a lot of tags:

<span style="color: blue;"><span style="color: blue;">[font='Times New Roman']<span style="font-size: 22pt;">THIS</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> IS </span>[/font]<span style="color: #FF3300;"><span style="color: #FF3300;">[font='Times New Roman']<span style="font-size: 22pt;">A TEST</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> USING </span>[/font]<span style="color: #00CC66;"><span style="color: #00CC66;">[font='Times New Roman']<span style="font-size: 22pt;">SOME</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> BEAUTIFUL </span>[/font]<span style="color: fuchsia;"><span style="color: fuchsia;">[font='Times New Roman']<span style="font-size: 22pt;">SOUP</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> | </span>[/font]<span style="color: #00CCFF;"><span style="color: #00CCFF;">[font='Times New Roman']<span style="font-size: 22pt;">96786</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> AND </span>[/font]<span style="color: #CC33FF;"><span style="color: #CC33FF;">[font='Times New Roman']<span style="font-size: 22pt;">HTML2TEXT</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> TO LEARN </span>[/font]<span style="color: red;"><span style="color: red;">[font='Times New Roman']<span style="font-size: 22pt;">NEW THING</span>[/font]</span></span>

Then i'm using html2text in order to get the raw text out of that chunk of code by:

h = html2text.HTML2Text()
h.ignore_links = True
h.ignore_images = True
h.ignore_emphasis = True
print h.handle(content) #content is that chunk of code

The best result I get so far is:

[font='Times New Roman']THIS[/font][font='Times New Roman'] THIS
[/font][font='Times New Roman']IS[/font][font='Times New
Roman'] A TEST [/font][font='Times New Roman']USING[/font][font='Times New
Roman'] BEAUTIFUL [/font][font='Times New Roman'] SOUP [/font][font='Times New Roman']
| [/font][font='Times New Roman']96786[/font][font='Times New Roman'] AND [/font][font='Times New Roman'] HTML2TEXT [/font][font='Times New Roman'] TO LEARN [/font][font='Times New Roman']NEW THING[/font]

How do I get rid of the [font] tag using html2text + beautifulsoup, or any other ways to do that? Thank you

My approach is im using string replace to replace [font ...] and [/font] with '' but that seem inefficient. Is it any other ways that we can solve it?

Answer 1

It looks like your input is a mix of HTML and BBCode. BeautifulSoup and html2text are both meant to parse & extract text from HTML, but not BBCode.

One simple solution would be to convert the [font] BBCode "tags" into HTML before processing with either BeautifulSoup or html2text. You could use regular expressions to do the conversion, see convert_bbcode_fonts below. (Note that this doesn't actually convert your input to "valid" HTML4 font tags, but html2text still handles the input.)

import re
import html2text


def convert_bbcode_fonts(html):
    flags = re.IGNORECASE | re.MULTILINE
    # replace start font tags
    html = re.sub(r'\[font\s*([^\]]+)\]', '<font \1>', html, flags=flags)
    # replace end font tags
    html = re.sub(r'\[/font\s*\]', '</font>', html, flags=flags)
    return html

def extract_text(html):
    html = convert_bbcode_fonts(html)
    h = html2text.HTML2Text()
    h.ignore_links = True
    h.ignore_images = True
    h.ignore_emphasis = True
    return h.handle(html)

INPUT = """
<span style="color: blue;"><span style="color: blue;">[font='Times New Roman']<span style="font-size: 22pt;">THIS</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> IS </span>[/font]<span style="color: #FF3300;"><span style="color: #FF3300;">[font='Times New Roman']<span style="font-size: 22pt;">A TEST</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> USING </span>[/font]<span style="color: #00CC66;"><span style="color: #00CC66;">[font='Times New Roman']<span style="font-size: 22pt;">SOME</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> BEAUTIFUL </span>[/font]<span style="color: fuchsia;"><span style="color: fuchsia;">[font='Times New Roman']<span style="font-size: 22pt;">SOUP</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> | </span>[/font]<span style="color: #00CCFF;"><span style="color: #00CCFF;">[font='Times New Roman']<span style="font-size: 22pt;">96786</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> AND </span>[/font]<span style="color: #CC33FF;"><span style="color: #CC33FF;">[font='Times New Roman']<span style="font-size: 22pt;">HTML2TEXT</span>[/font]</span></span>[font='Times New Roman']<span style="font-size: 22pt;"> TO LEARN </span>[/font]<span style="color: red;"><span style="color: red;">[font='Times New Roman']<span style="font-size: 22pt;">NEW THING</span>[/font]</span></span>
"""

if __name__ == '__main__':
    print extract_text(INPUT)