将未知字符串转换为Unicode-Python

Question

I want to use this dictionary file that is supposed to contain Japanese characters, but for some reason, it shows non-sense characters such as "乙 勹 月 田 亀" .

The introduction of the file states: "The encoding scheme now in use is no longer EUC-JP and the convenient 2 bytes for the JIS x 208 and 3 bytes for the JIS x 0212 . The encoding of this file is now UTF-8 , and as such, the byte length of each character is highly variable. Processing Unicode properly requires that your software does not rely on a fixed byte length. The primary reason for the change of encoding method is that the JIS x 0213 standard kanji are not defined in the Extended Unix Code Japanese encoding scheme which predates it ( EUC-JP )."

I tried without success to decode it using python 3:

unknown_string = "𪚲 : 乙 勹 月 田 亀" decoded_string = unknown_string.decode('unicode_escape').encode('latin-1').decode('utf8') print(decoded_string) (results in printing 𪚲 : ä¹™ 勹 月 ç”° 亀)

unknown_string = "𪚲 : 乙 勹 月 田 亀" decoded_string = unknown_string.encode('latin1').decode('utf-8') print(decoded_string) (results in UnicodeEncodeError: 'latin-1' codec can't encode character '\š' in position 2: ordinal not in range(256) )

I also tried looking at the bytes, but I see no connection. For instance, 化's hex value is \\xE5\\x8C\\x96 , but it is replaced in the file with 化 that has the value \\xC3\\xA5\\xC5\\x92\\xE2\\x80\\x93 .

How could I retrieve the original Japanese characters?

Answer 1

You can use this file from jpnetkit to prepare the dict:

krad = Kradfile()
krad.get_kradfile()
kanji_dict = krad.prepare_radikals()
for w in kanji_dict[u'㔟']:
    print w
# 丶 力 九 生

If you want to do the parse from scratch, you can check this link to get an unbroken version.

Or just download and unzip it. I think the reason is that the link should be downloaded, not opened in the browser. When the browser opens this gz file, it doesn't know the charset and thus displays non-sense characters.