使用Python检索img src属性的xpath语法有问题

Question

I've been trying to figure out the xpath syntax to parse this html, but I haven't been getting the same results as others. I've been modeling my work after, http://docs.python-guide.org/en/latest/scenarios/scrape/#web-scraping , but I can't get it to work for my html.

<div id="sku-8103">
    <!-- B:649 -->
    <input type="hidden" id="productIdPDP" value="1218866963585"/>
    <input type="hidden" id="skuIdPDP" value="8240103" />
    <input type="hidden" id="enableLightbox" value="" />
    <!-- B:780 -->
    <img src="http://images.bestbuy.com/BestBuy_US/en_US/images/global/buttons/btn_notorderable_pdp.gif" alt="Not Orderable" border="0" id="notorderable" />
    <input name="8240103" type="hidden" value="1">
    <!-- E:780 -->
    <!-- E:649 -->
    </div>

My Code:

import pycurl
import sys
import cStringIO
from lxml import etree
from lxml import html

buf = cStringIO.StringIO()

c = pycurl.Curl()
c.setopt(c.URL, 'http://www.bestbuy.com/site/sony-playstation-4-500gb/8240103.p?id=1218866963585&skuId=8240103')
c.setopt(c.WRITEFUNCTION, buf.write)
c.perform()

data = buf.getvalue()
buf.close()

tree = html.fromstring(data)


product = tree.xpath('//div[@id="sku-8240103"]/img[@src]')
print product

The output is: [] instead of the src value of the image. I also tried:

product = tree.xpath('//div[@id="sku-8240103"]/img[@src]/text()')

but that didn't seem to work either.

Answer 1

Your HTML has this:

<div id="sku-8103">

You're searching with this:

product = tree.xpath('//div[@id="sku-8240103"]/img[@src]')

Notice the different SKU number? There are no matching nodes, and therefore you get back the empty list, [] .

If you change it like this:

product = tree.xpath('//div[@id="sku-8103"]/img[@src]')

You now get a single-element list, like this:

[<Element img at 0x10c85b890>]

And if you do this:

print product[0].attrib['src']

… you get this:

http://images.bestbuy.com/BestBuy_US/en_US/images/global/buttons/btn_notorderable_pdp.gif

Really, you don't need the [@src] part there; if you're attempting to restrict it to img s that have a src attribute… what other img s do you expect to see?