Python: convert Chinese characters into pinyin with CJKLIB

Question

I'm trying to convert a bunch of Chinese characters into pinyin, reading the characters from one file and writing the pinyin into another. I'm working with the CJKLIB functions to do this.

Here's the code,

from cjklib.characterlookup import CharacterLookup

source_file = 'cities_test.txt'
dest_file = 'output.txt'

s = open(source_file, 'r')
d = open(dest_file, 'w')

cjk = CharacterLookup('T')

for line in s:
    p = line.split('\t')
    for p_shard in p:
        for c in p_shard:
            readings = cjk.getReadingForCharacter(c.encode('utf-8'), 'Pinyin')
            d.write(readings[0].encode('utf-8'))
        d.write('\t')
    d.write('\n')

s.close()
d.close()

My problem is that I keep running into Unicode-related errors, the error comes up when I call the getReadingForCharacter function. If I called it as written,

readings = cjk.getReadingForCharacter(c.encode('utf-8'), 'Pinyin')

I get: UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range (128).

If I call it like this, without the .encoding() ,

readings = cjk.getReadingForCharacter(c, 'Pinyin')

I get an error thrown by sqlalchemy (the CJKLIB uses sqlalchemy and sqlite): You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings ... etc.

Can someone help me out? Thanks!

Oh also, is there a way for CJKLIB to return the pinyin without any tones? I think by default it's returning pinyin with these weird characters to represent tones, I just want the letters without these tones.

Answer 1

Your bug is that you are not decoding the input stream, and yet you are turning around and re-encoding it as though it were UTF-8. That's going the wrong way.

You have two choices.

You can codecs.open the input file with an explicit encoding so you always get back regular Unicode strings whenever you read from it because the decoding is automatic. This is always my strong preference. There is no such thing as a text file anymore.

Your other choice is to manually decode your binary string it before you pass it to the function. I hate this style, because it almost always indicates that you're doing something wrong, and even when it doesn't, it is clumsy as all get out.

I would do the same thing for the output file. I just hate seeing manually .encode("utf-8") and .decode("utf-8") all over the place. Set the stream encoding and be done with it.