Unicode Decode Error in Python

Question

TDB = csv.reader(codecs.open('data/TDS.csv', 'rb', encoding='utf-8'), delimiter=',', quotechar='"')

ts = db.testCol

for row in TDB:
    print row[1]
    T = {"t":row[1],
             "s": row[0]}
    post_id = ts.insert(T)

I not sure why i can't encode it into utf-8 while i want to insert data into database i must make it in utf8 format.

UnicodeDecodeError: 'utf8' codec can't decode byte 0xf3 in position 36: invalid continuation byte

Before i put the encoding function, i got this from pymongo.

bson.errors.InvalidStringData: strings in documents must be valid UTF-8

and i guess, this is the data it couldn't encode

'compleja e intelectualmente retadora , el ladrÛn de orquÌdeas es uno de esos filmes que vale la pena ver precisamente por su originalidad . '

Anyone know how should i do? Thanks

Answer 1

Ok, this might help..

There are a list of encodings here:

http://docs.python.org/2/library/codecs.html#standard-encodings

latin-1 is a common encoding used for languages in europe.

The basic flow with dealing with encodings is:

1. read in encoded content
2. content.decode("source encoding") to unicode
3. encode from unicode to desired encoding, unicode_content.encode("desired encoding")

You can try going through encodings that seem right and see which ones don't cause an error:

enc = "latin-1"
f = open("TSD.csv", "r")
content = f.read() # raw encoded content
u_content = content.decode(enc) # decodes from enc to unicode
utf8_content = u_content.encode("utf8")