将 lxml.html 与损坏的 html 实体一起使用？

Question

I need to work with a page, which has an unfortunate mix of correct and incorrect HTML entities; for instance:

<i>Kristj&aacuten V&iacute;ctor</i>

This, in Firefox 67, does get interpreted correctly, eventually:

... however, if we do "View Source", Firefox indicates via syntax color that something is wrong with the first HTML entity:

... and indeed there is, a semicolon at the end of the HTML entity is missing - however, somehow Firefox figures it out, and renders the right character.

Now, if I try to work with that in lxml:

#!/usr/bin/env python3

import lxml.html as LH
import lxml.html.clean as LHclean

testhtmlstring = "<i>Kristj&aacuten V&iacute;ctor</i>"

myhtml = LH.fromstring( testhtmlstring )
myhtml = LHclean.clean_html( myhtml )
myitem = myhtml.xpath("//i")[0]
myitemstr = myitem.text_content()
print(myitemstr)

... the code prints out this in terminal (Ubuntu 18.04):

Kristj&aacuten Víctor

... so, obviously, the broken htmlentity did not get converted to the right character.

Is there something I can use, so I get the right character in my output string from lxml, even in case of a broken htmlentity (as Firefox does)?

Answer 1

The HTML 5 standard has specified a specific subset of entities that can be parsed without the trailing semicolon present, because these entities were historically defined with the semicolon being optional .

Thehtml.unescape() function explicitly supports those, use that function as a second pass to clear out this issue:

>>> from html import unescape
>>> unescape("Kristj&aacuten Víctor")
'Kristján Víctor'

If you install html5lib then you can have lxml behave the same, via their lxml.html.html5parser module (which wraps html5lib 's own html5lib.treebuilders.etree_lxml adapter ):

>>> from lxml.html import html5parser as etree
>>> etree.fromstring("Kristj&aacuten Víctor").text
'Kristján Víctor'