beautifulsoup无法正确解析html

Question

So I have the following code :

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from bs4 import BeautifulSoup

html = '</p></td></tr><tr><td colspan="3">   Data I want  </td></tr><tr>  <td colspan="3">   Data I want  </td> </tr> <tr><td colspan="3">   Data I want  </td> </tr></table>'
soup = BeautifulSoup(html, "lxml")

print soup.getText()

But the output is empty, yet with other html samples it works just fine. The html is like that because it is extracted from a table.

html = '<p>Content</p></td></table>'

That works just fine for example. Any help?

Edit: I know the HTML is not valid, but the second HTML sample is also invalid yet that works.

Answer 1

It's because lxml is having trouble parsing invalid HTML .

Use html.parser instead of lxml .

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from bs4 import BeautifulSoup

html = '</p></td></tr><tr><td colspan="3">   Data I want  </td></tr><tr>  <td colspan="3">   Data I want  </td> </tr> <tr><td colspan="3">   Data I want  </td> </tr></table>'
soup = BeautifulSoup(html, 'html.parser')

print soup.getText()

Output:

 Data I want      Data I want       Data I want   

Answer 2

if the consistent issue is missing the opening tag you can use regular expression to find what it should be like the below

from bs4 import BeautifulSoup
import re

html = '</p></td></tr><tr><td colspan="3">   Data I want  </td></tr><tr>  <td colspan="3">   Data I want  </td> </tr> <tr><td colspan="3">   Data I want  </td> </tr></table>'
pat = re.compile('</[a-z]*>')
L = list(re.findall(pat, html))
if L[0] != L[len(L)-1]:
    html = L[len(L)-1].replace('/','') + html

soup = BeautifulSoup(html, "lxml")
print soup.getText()

output is

Data I want      Data I want       Data I want 

Answer 3

What you have there is not a valid HTML. Why don't you change it to the following?

html = '<table><tr><td colspan="3">   Data I want  </td></tr><tr>  <td colspan="3">   Data I want  </td> </tr> <tr><td colspan="3">   Data I want  </td> </tr></table>'

But there is probably something missing before the sample you posted. Where does the HTML code come from?