Parsing html using lxml.html

Question

I'm developing a Python scraper at scraperwiki.com and I need to parse a html page that containt the following:

<div class="items">
  <div class="item">
       ItemLine1 ItemLine1 ItemLine1
       <br> 
       ItemLine2 ItemLine2 ItemLine2
 </div>
 <br>
</div>

What I'm doing now is:

import scraperwiki
import lxml.html

#.......................
raw_string = lxml.html.fromstring(scraperwiki.scrape(url_to_scrape))
my_line = ((raw_string.cssselect("div.items div.item")[0]).text)
print (my_line)

and it prints ItemLine1 ItemLine1 ItemLine1 only. When I change [0] to [1] it throws an exception.

How do I scrape that? Should I use xpath?

Answer 1

XPath is the most straightforward solution:

items = raw_string.cssselect('div.items div.item')

texts = [item.xpath('br[1]/preceding-sibling::node()') for item in items]

The XPath br[1] selects the first br child of div.item ; The preceding-sibling:: axis contains all nodes that occur before the first br ; node() selects every kind of node (text or elements) that are in that axis.

If your larger goal is to split up the children of a node by br elements, you can take a few different approaches. The reason this is so tricky is that elements like br and hr are badly designed markup. Using a tree-like markup language like sgml, html, or xml, things that should be together should be grouped by a common parent element rather than split by a childless delimiter element.

I'll expand your test case to demonstrate some more complex situations:

html = """<div class="items">
  <div class="item">
   <br>
   ItemLine1 ItemLine1 ItemLine1
   <a href="">item</a>
   Itemline1-b
   <br> 
   <a class="z">item2</a>
   ItemLine2 ItemLine2 ItemLine2
   <br><br>
   Itemline3
 </div>
 <br>
</div>"""

doc = lxml.html.fromstring(html)
itemlist = doc.cssselect('div.items div.item')

The first approach is to simply get all nodes in the paragraph and split them into different lists by br . If you use this approach, don't use the text and tail attributes of the ElementTree API because you will probably end up duplicating text.

def paras_by_br_nodes(parent):
    """Return a list of node children of parent (including text nodes) grouped by "paragraphs" which are delimited by <br/> elements."""
    paralist = []
    paras = []
    for node in parent.xpath('node()'):
        if getattr(node, 'tag', None) == 'br':
            paralist.append(paras)
            paras = []
        else:
            paras.append(node)
        paralist.append(paras)
        return paralist


print paras_by_br_nodes(itemlist[0])

This produces lists like so:

[['\n       '],
 ['\n       ItemLine1 ItemLine1 ItemLine1\n\t\t', <Element a at 0x10498a350>, '\n\t\tItemline1-b\n       '],
 [<Element a at 0x10498a230>, '\n       ItemLine2 ItemLine2 ItemLine2\n       '],
 [], 
 ['\n       Itemline3\n ']]

A second approach is to make use of the ElementTree API and keep text nodes in the text and tail attributes. The downside of this approach is that if there is no element to attach the text, we need to just include the text node. This list of non-homogenous types is a bit more trouble to work with.

def paras_by_br_text(parent):
    paralist=[]
    para=[parent.text]
    for item in parent:
        if item.tag=='br':
            paralist.append(para)
            para = [item.tail]
        else:
            para.append(item)
    paralist.append(para)
    return paralist

print paras_by_br_text(itemlist[0])

This produces a list like so. Note that in contrast to the previous list it only has text nodes nodes in the first position of the list. This corresponds to the br.tail text or parent.text (which is text before the first element).

[['\n       '],
 ['\n       ItemLine1 ItemLine1 ItemLine1\n\t\t', <Element a at 0x1042f5170>],
 [<Element a at 0x1042f5290>],
 [],
 ['\n       Itemline3\n ']]

I think the best approach is to introduce new elements. This html is using br when it should be using p or some other container element. So instead, let's fix the html and return a list of elements instead of a list of list of nodes:

def paras_by_br(parent):
    paralist = []
    para = lxml.html.etree.Element('para')
    if parent.text:
        para.text = parent.text
    for item in parent:
        if item.tag=='br':
            paralist.append(para)
            para = lxml.html.etree.Element('para')
            if item.tail:
                para.text = item.tail
        else:
            para.append(item)
    return paralist

paralist = paras_by_br(itemlist[0])

print "\n--------\n".join(lxml.html.etree.tostring(para) for para in paralist)

This prints the following:

<para>
       </para>
--------
<para>
       ItemLine1 ItemLine1 ItemLine1
        <a href="">item</a>
        Itemline1-b
       </para>
--------
<para><a class="z">item2</a>
       ItemLine2 ItemLine2 ItemLine2
       </para>
--------
<para/>

See how items are grouped by a new para element, which doesn't exist in the original document.