Python：UnicodeDecodeError：'ascii'编解码器无法解码位置0的字节0xd0：序数不在范围内（128）

Question

Scenario: I have a list of server names in a JSON file that is getting read by the script and put into a dictionary. I'm then trying to use those server names in what will become a SQL query. However, I'm having a hell of a time with the UTF-8 encoded strings.

Error Traceback:

Traceback (most recent call last):
  File "run.py", line 18, in <module>
    print(str(len(download.downloadRealmFiles('eu'))) + " EU files downloaded.")
  File "/var/www/etherealpost.com/scripts/ahdata/download.py", line 73, in downloadRealmFiles
    sql = u"UPDATE realms_lastmodified SET last_modified = '%d', latest_hash = '%s' WHERE region = '%s' AND realm = '%s'" % (lastModified, lastHash.encode('utf-8'), region.encode('utf-8'), realm)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

The code:

realm = data['files'][0]['realm']
lastHash = realmFile.split('/')[-2]
lastModified = data['files'][0]['lastModified']
dataURLs.append(realmFile)
sql = u"UPDATE realms_lastmodified SET last_modified = '%d', latest_hash = '%s' WHERE region = '%s' AND realm = '%s'" % (lastModified, lastHash.encode('utf-8'), region.encode('utf-8'), realm.encode('utf-8'))

lastModified is of type long The variable realm is the one that contains the Unicode characters.

I'm out of ideas why this isn't working.

Answer 1

Don't interpolate strings into a SQL query! Use SQL parameters instead and leave it up to your database to handle quoting and Unicode values:

sql = """\
    UPDATE realms_lastmodified
    SET last_modified=?, latest_hash=?
    WHERE region=? AND realm=?
"""
cursor.execute(sql, (lastModified, lastHash, region, realm))

I used ? as the parameter placeholders here, but it depends on the exact database library used; you may need to use %s as the placeholder instead ( regardless of the type of the column!).

Your error specifically is caused by you interpolating encoded bytestrings into a Unicode value. Don't do that either; interpolate, then encode. Otherwise, Python attempts to decode the UTF8 bytes using the default codec to get Unicode again, and that fails here.