Python：UnicodeEncodeError：'ascii'编解码器无法在位置34-39处编码字符：序数不在范围内（128）

Question

I've got a data of twitter log and I have to sort the file to show each user's retweeted tweet ranking.

Here's the code.

import codecs

with codecs.open('hoge_qdata.tsv', 'r', 'utf-8') as tweets:
    tweet_list = tweets.readlines()

tweet_list.pop(0)

---

facul={}
for t in tweet_list:
    t = t.split('\t')
    t[-2] = int(t[-2])   
    if t[-2] <= 0:      
        continue        
    if not t[0] in facul:
        facul[t[0]] = []
    facul[t[0]].append(t)

def cmp_retweet(a,b):
    if a[-2] < b[-2]:
        return 1
    if a[-2] > b[-2]:
        return -1
    return 0

for f in sorted(facul.keys()):
    facul[f].sort(cmp=cmp_retweet)
    print ('[%s]' %(f))
    for t in facul[f][:5]:
        print ('%d:%s:%s' % (t[-2], t[2], t[-1].strip())

Somehow I got an error saying:

print '%d:%s:%s' %(t[-2], t[2], t[-1].strip())
UnicodeEncodeError: 'ascii' codec can't encode characters in position
34-39: ordinal not in range(128)

Looks like Japanese language letters can't be decoded. How can I fix this? I tried to use sys.setdefaultencoding("utf-8") but then I got an error:

sys.setdefaultencoding("utf-8")
AttributeError: 'module' object has no attribute 'setdefaultencoding'

This is how I tried it:

import codecs
import sys
sys.setdefaultencoding("utf-8")

with codecs.open('hoge_qdata.tsv', 'r', 'utf-8') as tweets:
    tweet_list = tweets.readlines()

ps I am using Python version 2.7.5

Answer 1

The basic issue, as you have discovered, is that ASCII cannot represent much of unicode.

So you have to make a choice on how to handle it:

• don't display non-ASCII chars
• display non-ASCII chars as some other type of representation

The first choice would like this:

for t in facul[f][:5]:
    print ('%d:%s:%s' % (
            t[-2],
            t[2].encode('ascii', errors='ignore'),
            t[-1].encode('ascii', errors='ignore').strip()
            ))

While the second choice would replace ignore with something like replace , xmlcharrefreplace , or backslashreplace .

Here's the reference .

Answer 2

The error message is giving you two clues: first, the problem is in the statement

print '%d:%s:%s' %(t[-2], t[2], t[-1].strip())

Second, the problem is related to an encode operation. If you don't remember what is meant by "encode", now would be a good time to re-read the Unicode HOWTO in the Python 2.7 docs.

It looks like your list t[] contains Unicode strings. The print() statement is emitting byte strings. The conversion of Unicode strings to byte strings is encoding . Because you aren't specifying an encoding, Python is implicitly doing a default encoding. It uses the ascii codec, which cannot handle any accented or non-Latin characters.

Try splitting that print() statement into two parts. First, insert the unicode t[] values into a unicode format string. Note the use of u'' syntax. Second, encode the unicode string to UTF and print.

s = u'%d:%s:%s' %(t[-2], t[2], t[-1].strip())
print s.encode('utf8')

(I haven't tested this change to your code. Let me know if it doesn't work.)

I think sys.setdefaultencoding() is probably a red herring, but I don't know your environment well.

By the way, the statement, as you write it above, has unbalanced parentheses. Did you drop a right parenthesis when you pasted in the code?

print ('%d:%s:%s' %(t[-2], t[2], t[-1].strip())