Using python3, what is the fastest way to extract all <div> blocks from a html str?

Question

there are <div> inner blocks inside a <div> block, What is the fastest way to extract all <div> blocks from a html str ? (bs4, lxml or regex ?)

Answer 1

lxml is generally considered to be the fastest among existing Python parsers, though the parsing speed depends on multiple factors starting with the specific HTML to parse and ending with the computational power you have available. For HTML parsing use the lxml.html subpackage:

from lxml.html import fromstring, tostring

data = """my HTML string"""
root = fromstring(data)

print([tostring(div) for div in root.xpath(".//div")]) 
print([div.text_content() for div in root.xpath(".//div")]) 

There is also the awesome BeautifulSoup parser which, if allowed to use lxml under-the-hood , would be a great combination of convenience, flexibility and speed. It would not be generally faster than pure lxml , but it comes with one of the best APIs I've ever seen allowing you to "view" your XML/HTML from different angles and use a huge variety of techniques:

from bs4 import BeautifulSoup

soup = BeautifulSoup(data, "lxml")
print([str(div) for div in soup.find_all("div")])
print([div.get_text() for div in soup.find_all("div")])

And, I personally think, there is rarely a case when regex is suitable for HTML parsing:

• RegEx match open tags except XHTML self-contained tags

Answer 2

When I'm teaching XML/HTML parsing with Python, I use to show this levels of complexity:

1. RegEx: efficient for (very) simple parsing but can be/become hard to maintain.
2. SAX: efficient and safe to parse XML as a stream. Easy to extract pieces of data but awful when you want to transform the tree. Can become really difficult to maintain. Who still use that anyway?
3. DOM parsing or ElementTree parsing with lxml : less efficient: all the XML tree is loaded in memory (can be an issue for big XML). But this library is compiled (in Cython). Very popular and reliable. Easy to understand: the code can be maintained.
4. XSLT1 is also a possibility. Very good to transform the tree in depth. But not efficient because of the templates machinery. Need learn a new language which appears to be difficult to learn. Maintenance often become heavy. Note that lxml can do XSLT with Python functions as an extension.
5. XSLT2 is very powerful but the only implementation I know is in Java language with Saxon. Launching the JRE is time consuming. The language is difficult to learn. One need to be an expert to understand every subtleties. Worse as XSLT1.

For your problem, lxml (or BeautifulSoup ) sound good.