Unicode相等比较失败的Python屏幕抓取Twitter页面

Question

我正在使用以下代码在Twitter上获取用户关注者的列表：

import urllib
from BeautifulSoup import BeautifulSoup

#code only looks at one page of followers instead of continuing to all of a user's followers
#decided to only use a small sample 

site = "http://mobile.twitter.com/NYTimesKrugman/following"
friends = set()
response = urllib.urlopen(site)
html = response.read()
soup = BeautifulSoup(html)
names = soup.findAll('a', {'href': True})
for name in names:
    a = name.renderContents()
    b = a.lower()
    if ("http://mobile.twitter.com/" + b) == name['href']:
        c = str (b)
        friends.add(c)

for friend in friends:
    print friend
print ("Done!")

但是，我得到以下结果：

NYTimeskrugman
nytimesphoto
rasermus

Warning (from warnings module):
   File "C:\Users\Public\Documents\Columbia Job\Python Crawler\Twitter     Crawler\crawlerversion14.py", line 42
    if ("http://mobile.twitter.com/" + b) == name['href']:
 UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
amnesty_norge
zynne_
fredssenteret
oljestudentene
solistkoret

....（因此继续）

看来我能够获得以下大多数名称，但收到了一些随机错误。 它并没有阻止代码完成，但是...我希望有人能启发我发生了什么事？

Answer 1

不知道几年后我的答案是否会有用，但是我使用请求而不是urllib重写了您的代码。

我认为最好使用“用户名”类进行其他选择，以仅考虑关注者名称！

这是东西：

import requests
from bs4 import BeautifulSoup

site = "http://mobile.twitter.com/paulkrugman/followers"
friends = set()
response = requests.get(site)
soup = BeautifulSoup(response.text)
names = soup.findAll('a', {'href': True})
for name in names:
    pseudo = name.find("span", {"class": "username"})
    if pseudo:
        pseudo = pseudo.get_text()
        friends.add(pseudo)

for friend in friends:
    print (friend)
print("Done !")

@paulkrugman出现在每个集合中，所以不要忘记删除它！