将字符转换为unicode

Question

This code is working as expected. Only problem is that if there is unicode character, it gets converted to ASCII.

with open('test.idx', 'w') as writefile:
    with open('test.dat') as myfile:
        mystr=myfile.read()
        for myword in mystr.split('|'):
            tow=myword, '|', mystr.index(myword)
            print >>writefile, tow

In [74]: !cat test.dat
UTF-8
जनन|1
जन्म देणे
शिक्षण|1
 क्षेत्रातील संशोधनाच्या बाजारीकरणा बाबतीत व्यक्त केलेली 
पूर्व|1
 पगड्यामुळे

In [75]: !cat test.idx
('UTF-8\n\xe0\xa4\x9c\xe0\xa4\xa8\xe0\xa4\xa8', '|', 0)

I expect to see unicode instead of escaped code.

Answer 1

You created a tuple:

tow=myword, '|', mystr.index(myword)

That's not a string object, that's a tuple containing three other objects, two of which are strings, one an integer.

When you then write that tuple to the file, Python has to convert it to a string. Converting any Python container (be that a tuple, a list, a set or a dictionary) will use the repr() representation of the contained objects. For strings, that means only printable ASCII characters are allowed and shown, everything else uses escape sequences, most often the \\xhh form.

If that is not correct output for your usecase, you need to do the string conversion yourself. You could use string formatting:

tow = '{}|{}'.format(myword, mystr.index(myword))

If you are producing a lot of | -separated data, you may want to look at the csv module instead to handle the separator and file writing.

Answer 2

You are seeing the repr representation as you are storing the data in a tuple. To match your expected output use str.join:

     print >>writefile, "".join(map(str,tow))

The output file will contain:

UTF-8
जनन|0
1
जन्म देणे
शिक्षण|16
1
 क्षेत्रातील संशोधनाच्या बाजारीकरणा बाबतीत व्यक्त केलेली 
पूर्व|63
1
 पगड्यामुळे|239

If you add a print(tow) in your code you will see you have tuples.

('UTF-8\n\xe0\xa4\x9c\xe0\xa4\xa8\xe0\xa4\xa8', '|', 0)
('1\n\xe0\xa4\x9c\xe0\xa4\xa8\xe0\xa5\x8d\xe0\xa4\xae \xe0\xa4\xa6\xe0\xa5\x87\xe0\xa4\xa3\xe0\xa5\x87\n\xe0\xa4\xb6\xe0\xa4\xbf\xe0\xa4\x95\xe0\xa5\x8d\xe0\xa4\xb7\xe0\xa4\xa3', '|', 16)
('1\n \xe0\xa4\x95\xe0\xa5\x8d\xe0\xa4\xb7\xe0\xa5\x87\xe0\xa4\xa4\xe0\xa5\x8d\xe0\xa4\xb0\xe0\xa4\xbe\xe0\xa4\xa4\xe0\xa5\x80\xe0\xa4\xb2 \xe0\xa4\xb8\xe0\xa4\x82\xe0\xa4\xb6\xe0\xa5\x8b\xe0\xa4\xa7\xe0\xa4\xa8\xe0\xa4\xbe\xe0\xa4\x9a\xe0\xa5\x8d\xe0\xa4\xaf\xe0\xa4\xbe \xe0\xa4\xac\xe0\xa4\xbe\xe0\xa4\x9c\xe0\xa4\xbe\xe0\xa4\xb0\xe0\xa5\x80\xe0\xa4\x95\xe0\xa4\xb0\xe0\xa4\xa3\xe0\xa4\xbe \xe0\xa4\xac\xe0\xa4\xbe\xe0\xa4\xac\xe0\xa4\xa4\xe0\xa5\x80\xe0\xa4\xa4 \xe0\xa4\xb5\xe0\xa5\x8d\xe0\xa4\xaf\xe0\xa4\x95\xe0\xa5\x8d\xe0\xa4\xa4 \xe0\xa4\x95\xe0\xa5\x87\xe0\xa4\xb2\xe0\xa5\x87\xe0\xa4\xb2\xe0\xa5\x80 \n\xe0\xa4\xaa\xe0\xa5\x82\xe0\xa4\xb0\xe0\xa5\x8d\xe0\xa4\xb5', '|', 63)
('1\n \xe0\xa4\xaa\xe0\xa4\x97\xe0\xa4\xa1\xe0\xa5\x8d\xe0\xa4\xaf\xe0\xa4\xbe\xe0\xa4\xae\xe0\xa5\x81\xe0\xa4\xb3\xe0\xa5\x87', '|', 239)

You also have utf-8 encoded strings not unicode, if you printed the individual elements from tow you would also see the correct output.