exceptions.UnicodeDecodeError - 'ascii' codec can't decode byte

Question

I keep getting this error:

<type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
      args = ('ascii', '\xe2\x9d\xb6 Senn =)', 0, 1, 'ordinal not in range(128)')
      encoding = 'ascii'
      end = 1
      message = ''
      object = '\xe2\x9d\xb6 Senn =)'
      reason = 'ordinal not in range(128)'
      start = 0

Using this code:

    steamFriend = data['response']['players'][i]
    n = steamUser(steamFriend['personaname'].encode("utf-8"), steamFriend['steamid'], steamFriend['avatarfull'], steamFriend['profileurl'], steamFriend['personastate'], False)

Some things to note here:

• steamFriend is a JSON object
• I get this error only sometimes, because the steamFriend['personaname'] contains some weird symbols (for example ❶), and I don't know how to parse this correctly so I don't get errors.

Any help is greatly appreciated. Also, \\xe2\\x9d\\xb6 Senn =) is supposed to represent ❶ Senn =) , if that helps.

Answer 1

Without seeing the full code it is hard to tell, but it seems that steamUser expects ascii input. If that is the problem, you can solve it by:

streamFriend['personaname'].encode("ascii", errors="ignore")

or

streamFriend['personaname'].encode("ascii", errors="replace")

Obviously you will lose unicode characters in the process.

Answer 2

If the quoted error is occurring on the n=... line, the implication is that steamFriend['personaname'] is a byte string, not a Unicode string.

Consequently when you ask to .encode it, Python has to de code the string to Unicode in order to be able to en code it back to bytes. An implicit decoding happens using the default encoding, which is ASCII, so because the byte string does not contain only ASCII you get a failure.

Are you sure you didn't mean to do:

steamFriend['personaname'].decode("utf-8")

de coding the byte string '\\xe2\\x9d\\xb6 Senn =)' using UTF-8 would give you the Unicode string u'\❶ Senn =)' , where U+2776=❶ so that would seem more like what you are after.

(Normally, however, JSON strings are explicitly Unicode, so it's not clear where you would have got the byte string from. How are you loading the JSON content?)