unicode, not str in python

Question

I tried to run this Python code:

with io.open(outfile, 'w' ) as processed_text, io.open(infile, 'r') as fin:
    for line in fin:
        processed_text.write(preprocess(line.rstrip())+'\n')

but got TypeError: must be unicode, not str

How can i solve this problem? I searched here for similar problems and found one to try like

with io.open(outfile, 'w', encoding="utf-8") as processed_text, io.open(infile, 'r') as fin:

but didn't work.

Answer 1

Try putting this at the very top of your file:

from __future__ import unicode_literals

Python 3.x uses unicode by default. This will cause Python 2.x to follow the same behavior.

If you still have issues you can manually cast the problem string ala

uni_string = unicode(my_string)

Answer 2

Make sure you write a unicode string when opening a file using io.open . Something like this should do the trick:

with io.open(outfile, 'w' ) as processed_text, io.open(infile, 'r') as fin:
    for line in fin:
        s = preprocess(line.rstrip())
        if isinstance(s, str):
            s = s.decode('utf8')
        processed_text.write(s + u'\n')

Or modify preprocess to make sure it returns a unicode string.

Answer 3

尝试在已处理的字符串前面写u，例如[u'blah']

Answer 4

Note :

Since this module has been designed primarily for Python 3.x, you have to be aware that all uses of “bytes” in this document refer to the str type (of which bytes is an alias), and all uses of “text” refer to the unicode type. Furthermore, those two types are not interchangeable in the io APIs.

In [1]: import io

In [2]: def preprocess(s):
   ...:     return bytes(s)
   ...: 

In [3]: with io.open('tst1.out', 'w') as processed_text, io.open('tst1', 'r') as fin:
   ...:     for line in fin:
   ...:         try:
   ...:             out_line = unicode(preprocess(line.rstrip() + '\n'), 'utf-8')
   ...:         except TypeError:
   ...:             out_line = preprocess(line.rstrip() + '\n')
   ...:         processed_text.write(out_line)