URL的Python XPath返回空列表

Question

I am trying to access an element of the following URL using XPath: http://www.booking.com/searchresults.html?dest_id=2400&dest_type=region&offset=288

The specific element I am looking for is the Div class "sr_item_link_to_villas ". I have been using the following XPath to trying to access it (in this example I am trying to access the second listing, but the full script loops through each listing), but it returns an empty list:

//*[@id="hotellist_inner"]/*[contains(@class,"sr_item")][2]//*[contains(@class,"sr_item_link_to_villas ")]

The full code is:

url='http://www.booking.com/searchresults.html?dest_id=2400&dest_type=region&offset=288'
page = parse(url).getroot()
pathstr='//*[@id="hotellist_inner"]/*[contains(@class,"sr_item")][2]//*[contains(@class,"sr_item_link_to_villas ")]'
content=page.xpath(pathstr)

Answer 1

Following Code may solve your purpose. You have to add header value for get data.

    import urllib2
    from lxml import etree
    from lxml.html import tostring,fromstring

    def get_HTML(url):
        header={"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0","Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Connection": "keep-alive"}
        req=urllib2.Request(url,None,header)
        return urllib2.urlopen(req).read()

    url="http://www.booking.com/searchresults.html?dest_id=2400&dest_type=region&offset=288"

    read = get_HTML(url)
    tree = etree.HTML(read)    
    data = tree.xpath("//div[@class='sr_item_link_to_villas ']/a/text()");
    print data