python beautifulsoup : lxml html.parser

Question

I must use beautifulsoup, but i don't know which parser I have to take. I hesitate between lxml and html.parser, or why not both. How to know if a web page is lxml compliant ? How to know if a web page is html parser compliant ? Many thanks

Answer 1

There is no silver bullet. Different HTML parsers behave differently and you should pick the one that works for your particular page. Works in this case basically means, that you can get to your desired data.

lxml parser is generally faster, html5lib is the most lenient one - this kind of difference would be relevant if you have a broken or non-well-formed HTML to parse. html.parser is built-in and can help to avoid extra dependencies, if this is a problem. Here is a related table that highlights the differences.

Answer 2

I've learned it the hard way. It's been killing me. I just couldn't figure out why the tag I wanted included something that wasn't in that tag. Turned out the html parser wasn't working correctly with that site. After hours of headache, I suddenly tried switching to lxml parser, and lo and behold... The unwated stuff was gone as it should have been!