让 BeautifulSoup 像浏览器一样处理换行符

Question

I'm using BeautifulSoup (version '4.3.2' with Python 3.4) to convert html documents to text. The problem I'm having is that sometimes web pages have newline characters "\\n" that wouldn't actually get rendered as a new line in a browser, but when BeautifulSoup converts them to text, it leaves in the "\\n".

Example:

Your browser probably renders the following all in one line (even though have a newline character in the middle):

This is a paragraph.

And your browser probably renders the following in multiple lines even though I'm entering it with no newlines:

This is a paragraph.

This is another paragraph.

But when BeautifulSoup converts the same strings to text, the only line line breaks it uses are the newline literals - and it always uses them:

from bs4 import BeautifulSoup

doc = "<p>This is a\nparagraph.</p>"
soup = BeautifulSoup(doc)

soup.text
Out[181]: 'This is a \n paragraph.'

doc = "<p>This is a paragraph.</p><p>This is another paragraph.</p>"
soup = BeautifulSoup(doc)

soup.text
Out[187]: 'This is a paragraph.This is another paragraph.'

Does anyone know how to make BeautifulSoup extract text in a more beautiful way (or really just get all the newlines correct)? Are there any other simple ways around the problem?

Answer 1

get_text might be helpful here:

>>> from bs4 import BeautifulSoup
>>> doc = "<p>This is a paragraph.</p><p>This is another paragraph.</p>"
>>> soup = BeautifulSoup(doc)
>>> soup.get_text(separator="\n")
u'This is a paragraph.\nThis is another paragraph.'

Answer 2

While I do realize this is an old post, I wanted to highlight some behavior in bs4 in the way text is printed from tags.

I'm not an html expert but these are the few things I considered while trying to make bs4 print text as a browser would -

The behaviors I'm about to describe are applicable to tag.get_text() and tag.find_all(text=True,recursive=True) functionalities in BeautifulSoup

1) New Lines in the HTML Source

Beautiful soup prints a new line if it is available in the html source

2) Implicit new lines due to block level elements

Beautiful soup does not add new lines before and after block elements like 'p' if there are no source new lines around the tag

3) <br> tags

BeautifulSoup does not print a new line if the source contains a <br> tag and there are no source new lines around the <br> tag

Solution

Here is a solution that works for many cases (the limiting factor being -1)The list of all inline elements 2) How CSS/JS might affect the inline-ness or block-ness at runtime in a browser environment

def get_text(tag:bs4.Tag) -> str:
    _inline_elements = {"a","span","em","strong","u","i","font","mark","label","s","sub","sup","tt","bdo","button","cite","del","b","a","font"}

    def _get_text(tag:bs4.Tag) -> Generator:
     
        for child in tag.children:
            if type(child) is Tag:
                # if the tag is a block type tag then yield new lines before after
                is_block_element = child.name not in _inline_elements
                if is_block_element: yield "\n"
                yield from ["\n"] if child.name=="br" else  _get_text(child)
                if is_block_element: yield "\n"
            elif type(child) is NavigableString:
                yield child.string
     return "".join(_get_text(tag))

Answer 3

I use the following small library to accomplish this:

https://github.com/TeamHG-Memex/html-text

pip install html-text

As simple as:

>>> import html_text
>>> html_text.extract_text('<h1>Hello</h1> world!')
'Hello\n\nworld!'

Answer 4

I would take a look at python-markdownify . It turns html into pretty readable text in markdown format.

It is available at pypi : https://pypi.python.org/pypi/markdownify/0.4.0

and github : https://github.com/matthewwithanm/python-markdownify