UnicodeEncodeError: 'ascii' codec can't encode character u'\u05a0' in position 34: ordinal not in range(128)

Question

I'm Trying to run a piece of code in python where I have to read a filename.txt file which contains Json format code. But I have some Unicode values in json values. The file is very Large but I have found One Unicode in file as ֠ this symbol whose unicode for Python is u"\֠"

You can Refer this link for More information on unicode Unicode Character 'HEBREW ACCENT TELISHA GEDOLA' (U+05A0)

my Python Code Look like

import MySQLdb
import json



db = MySQLdb.connect(host="10.233.188.84",    # your host, usually localhost
                 user="root",         # your username
                 passwd="freebird@123",  # your password
                 db="Practice_For_Json",)        # name of the data base


#cursor = db.cursor()
json_file = open('asda.txt', 'r' )
file_data = json.load(json_file)
print(file_data)
print(type(file_data))

datas = file_data['datads']
print(datas)
for data in datas:
    ex_statement = "Insert into `tablename` values {first_col '"+str(data['first_col'])+"'}, {second_col  '"+str(data['second_col'])+"'});"
#    cursor.execute(ex_statement)


db.close()

My Json Look Like:

{"datads" :[{
      "first_col" : "SoomeVAlue_1",
      "second_col" : "SomeValue_1_1"
},
{
     "first_col" : " Unicode_Start ֠  Unicode_End",
     "second_col" : "SomeValue_2_2"
}]}

output of above code is :

{u'datads': [{u'first_col': u'SoomeVAlue_1', u'second_col': u'SomeValue_1_1'}, {u'first_col': u' Unicode_Start \u05a0  Unicode_End', u'second_col': u'SomeValue_2_2'}]}
<type 'dict'>
[{u'first_col': u'SoomeVAlue_1', u'second_col': u'SomeValue_1_1'}, {u'first_col': u' Unicode_Start \u05a0  Unicode_End', u'second_col': u'SomeValue_2_2'}]
Traceback (most recent call last):
  File "abc.py", line 21, in <module>
    ex_statement = "Insert into `tablename` values {first_col '"+str(data['first_col'])+"'}, {second_col  '"+str(data['second_col'])+"'});"




UnicodeEncodeError: 'ascii' codec can't encode character u'\u05a0' in position 15: ordinal not in range(128)

when I'm Running this code I'm getting error as the title.
I'm Using Python 2.7 In SSH shell.
Please Help me with this.

Answer 1

When processing unicode in Python2 it's important to ensure all strings are unicode strings, otherwise there will be problems. This line is therefore problematic:

ex_statement = "Insert into `tablename` values {first_col '"+str(file_data['first_col'])+"'}, {second_col file_data '"+str(['first_col'])+"'});"

Calling str on will a unicode object cause a UnicodeEncodeError if the unicode uncludes non-ascii characters. So

str(file_data['first_col'])

should be

unicode(file_data['first_col'])

To avoid Python potentially corrupting the final string, all the string literals should be made unicode literals by prefixing them with u , for example

u"Insert into `tablename` values {first_col '"

These steps will ensure that your statement is unicode. Depending on your database's configuration a unicode statement may work, or you may need to encode the statement to whatever encoding the database requires.

Finally, manually creating statements like this is unsafe and can be difficult to get right - look into parameter substitution . A properly constructed statement might look like this:

ex_statement = u"INSERT INTO MYTABLE (col1, col2) VALUES (%s, %s)"
cursor.execute(ex_statement, (u'foo', u'bar'))

Answer 2

I think you can try this,

json_file = open('test.txt','rb')
json_file = json_file.read().decode('UTF-8')