使用LXML从网页中获取特定值

Question

I am trying to grab a value from a page using LXML and Python.

I followed some basic examples which worked. But I'm struggling to get the text from quite a complex (to me at least) web page.

I want to grab the number of followers from this page: http://twitter.com/aberdeencc

I want the exact value of followers which (at the time of writing is 10,623 - not the displayed 10.6K. The exact value is just shown as a tooltip-style mouseover.

Looking at the page code it is in this section:

<a class="ProfileNav-stat ProfileNav-stat--link u-borderUserColor u-textCenter js-tooltip js-openSignupDialog js-nonNavigable u-textUserColor" data-nav="followers" 
   href="/AberdeenCC/followers" data-original-title="10,623 Followers">
       <span class="ProfileNav-label">Followers</span>
       <span class="ProfileNav-value" data-is-compact="true">10.6K</span>
</a>

The code I have is

from lxml import html

import requests

page = requests.get('http://twitter.com/aberdeencc')

tree = html.fromstring(page.text)

followers = tree.xpath('//span[@class="ProfileNav-stat ProfileNav-stat--link 

u-borderUserColor u-textCenter js-tooltip js-openSignupDialog js-nonNavigable 

u-textUserColor"]/text()')

print 'Followers: ', followers

But that returns an empty list.

(I know a don't need a list for a single value, but I'm working from existing code)

Thanks for any pointers you can give

Watty

Answer 1

>>> from lxml import etree
>>> import requests
>>> page = requests.get("https://twitter.com/aberdeencc")
>>> doc = etree.HTML(page.text)
>>> doc.xpath('//a[@data-nav="followers"]/@title')
['10,623 Followers']

Answer 2

I'd advise agains using xpath in that particular case. I think the CSS selector API is better suited for that case. This should work:

followers = tree.cssselect("a.ProfileNav-stat")[0].attrib["data-original-title"]
# followers = '10,623 Followers'

This method requires cssselect to be installed.

Answer 3

I'd rely on the data-nav attribute instead and get the value of the title attribute:

from lxml import html
import requests


page = requests.get('http://twitter.com/aberdeencc')
tree = html.fromstring(page.text)

followers = tree.xpath('//a[@data-nav="followers"]/@title')
print 'Followers: ', followers

Prints:

Followers:  ['10,623 Followers']

In order, to exract the actual number from the followers , you can use a regular expression and then parse the string to int using locale.atoi() :

import locale
import re
from lxml import html
import requests


locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

page = requests.get('http://twitter.com/aberdeencc')
tree = html.fromstring(page.text)

followers = tree.xpath('//a[@data-nav="followers"]/@title')[0]
followers = re.match(r'^([0-9,]+)\sFollowers$', followers).group(1)
followers = locale.atoi(followers)

print 'Followers:', int(followers)

Prints:

Followers: 10623

---

Besides, twitter provides an API which you can use through the python interface, there are multiple options to choose from:

• tweepy
• python-twitter
• twitter