UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 54: ordinal not in range(128)

Question

I know, it has been asked several times, but none of the answer gave me a solution

Here is the code (Python 2.7):

import cx_Oracle
import pandas as pd

connstr = 'MyConstr'
conn = cx_Oracle.connect(connstr)
cur = conn.cursor()

xl = pd.ExcelFile("C:\\TEMP\\for_kkod.xlsx")
df = xl.parse(0)

for i in df.index:
    s = u"insert into MY_TABLE values({0}, '{1}')".format(int(df.iloc[i]['kkod']), df.iloc[i]['kkodnev'])
    print s
    print type(s)
    cur.execute(s)

The result of the 2 prints are this:

insert into MY_TABLE values(10, 'Készítés')
<type 'unicode'>

As you can see the type of s is unicode but nevertheless I have this error message:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 54: ordinal not in range(128)

I have tried with and without u"", with and without encode and decode in all possible ways but still the same error message

Any idea?

Answer 1

You are feeding a Unicode SQL statement to cursor.execute() . That method can only take a bytestring SQL statement .

You should not be using string interpolation to insert your Unicode values into the SQL query (which itself is just ASCII). Use query parameters, always!

 s = "insert into MY_TABLE values(:0, :1)"
 cur.execute(s, (int(df.iloc[i]['kkod']), df.iloc[i]['kkodnev']))

Now the values to be inserted are passed in as parameters and it is up to the database adapter to worry about encoding these correcty (as well as properly escaping the values to evade SQL injection issues).

The above uses numbered (positional) arguments, you can also use named parameters, pass in the values in a dictionary with matching keys:

 s = "insert into MY_TABLE values(:kkod, :kkodnev)"
 cur.execute(s, {'kkod': int(df.iloc[i]['kkod']), 'kkodnev': df.iloc[i]['kkodnev']})

You do have to make sure both your connection and your table column is correctly configured to handle Unicode. For example, you'll have to set the NLS_LANG option:

import os
os.environ['NLS_LANG'] = '.AL32UTF8'

Answer 2

Simply use the following parameters for connection:

connection = cx_Oracle.connect(connectString, encoding="UTF-8",nencoding="UTF-8")