使用lxml从html提取属性

Question

I use lxml to retrieve the attributes of tags from an html page. The html page is formatted like this:

<div class="my_div">
    <a href="/foobar">
        <img src="my_img.png">
    </a>
</div>

The python script I use to retrieve the url inside the <a> tag and the src value of the <img> tag inside the same <div> , is this:

from lxml import html 

...
tree = html.fromstring(page.text)
for element in tree.xpath('//div[contains(@class, "my_div")]//a'):
    href = element.xpath('/@href')
    src = element.xpath('//img/@src')

Why don't I get the strings?

Answer 1

You are using lxml so you are operating with lxml objects - HtmlElement instances. HtmlElement is nested from etree.Element: http://lxml.de/api/lxml.etree._Element-class.html , it have get method, that returns attrubute value. So the proper way for you is:

from lxml import html 

...
tree = html.fromstring(page.text)
for link_element in tree.xpath('//div[contains(@class, "my_div")]//a'):
    href = link_element.get('href')
    image_element = href.find('img')
    if image_element:
        img_src = image_element.get('src') 

Answer 2

If you change your code to:

from lxml import html 

...
tree = html.fromstring(page.text)
for element in tree.xpath('//div[contains(@class, "my_div")]//a'):
    href = element.items()[0][1]  #gives you the value corresponding to the key "href"
    src = element.xpath('//img/@src')[0]
    print(href, src)

You'll get what you need.

The documentation of lxml mentions some of these things, but I feel it is lacking a few things and you might want to consider using an interactive python shell to study the properties of the instances returned by tree.xpath() . Or you could look into another parser completely, such as BeautifulSoup , which has very good examples and documentation. Just sharing...

Answer 3

The reason why you didn't get the results you want is because you're trying to get attributes from the NEXT children rather than the existing node.

See this:

from lxml import html

s = '''<div class="my_div">
    <a href="/foobar">
        <img src="my_img.png">
    </a>
</div>'''

tree = html.fromstring(s)

# when you do path... //a, you are ALREADY at 'a' node
for el in tree.xpath('//div[contains(@class, "my_div")]//a'):
    # you were trying to get next children /@href, which doesn't exist
    print el.xpath('@href') # you should instead access the existing node's 
    print el.xpath('img/@src') # same here, not /img/@src ...

['/foobar']
['my_img.png']

Hope this helps.