使用python beautiful soup和requests包时，HTML内容不正确

Question

The HTML content I get after parsing a webpage using JSoup and BeautifulSoup is different as seen below. Does anyone have the same issue and can you please let me know what was done to fix this?

Check the third line in each block -

======= JSoup

<div class="col-full">
 <p><strong>Index Notifications</strong></p>
 <p></p><br>
<p> <br /> <b> March 28, 2014</b>
<br >
<br >

======= BeautifulSoup

<div class="col-full">
<p><strong>Index Notifications</strong></p>
<p><p> <br>
<b> March 28, 2014</b>
<br>
<br>

Answer 1

When parsing broken HTML, different parsers will try to repair the broken tags differently; there are no hard and fast rules on how to handle such errors.

BeautifulSoup can make use of different parsers , and each will handle your content differently:

>>> import requests
>>> from bs4 import BeautifulSoup
>>> url = 'http://www.wisdomtree.com/etfs/index-notices.aspx'
>>> html = requests.get(url).content
>>> BeautifulSoup(html, 'html.parser').find('div', class_='col-full')
<div class="col-full">
<p><strong>Index Notifications</strong></p>
<p><p> <br>
<b> March 28, 2014</b>
<br> <br>
# ... cut ...
>>> BeautifulSoup(html, 'lxml').find('div', class_='col-full')
<div class="col-full">
<p><strong>Index Notifications</strong></p>
<p></p><p> <br/>
<b> March 28, 2014</b>
<br/> <br/>
# ... cut ...
>>> BeautifulSoup(html, 'html5lib').find('div', class_='col-full')
<div class="col-full">

            <p><strong>Index Notifications</strong></p>
            <p></p><p> <br/>
<b> March 28, 2014</b>
<br/>  <br/>
# ... cut ...

The html5lib parser is the slowest, but will generally parse broken HTML exactly like most browsers would. Both lxml and html5lib parsed this specific section of the document pretty much like JSoup did.