python无法编码奇怪的字符

Question

I am attempting to parse txt file > 2GB by running following script:

#!/usr/bin/env python
import json

def convert2json(filename):
    with open(filename) as I:
        for line in I:
            d = {"data": line}
            print(json.dumps(d, ensure_ascii=False))

if __name__ == "__main__":
    import sys

    convert2json(sys.argv[1])

Script throws error:

Traceback (most recent call last):
  File "ori.py", line 13, in <module>
    convert2json(sys.argv[1])
  File "ori.py", line 8, in convert2json
    print(json.dumps(d))
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 31: invalid continuation byte

and it fails (I believe) when processing special characters:

<E0>ทำคาม:
:<E9>皇甫
:<E9>皇甫:<E9>皇甫:<E9>皇甫

How can I make the script to just ignore the lines that are causing problems?

When I go to the file that I am parsing and copy a big chunk of the lines that cannot be processed, create a new file and run the script again - than it works. I am doing that by copying lines from less and than to a file using vi . Am I doing something when copying the lines with the encoding itself?

Answer 1

Ok, you are using Python 2, so what you read from the file is a byte string. More according to the error message , you have the default for ensure_ascii parameter which is true. In that case, all strings are decoded with the default encoding (utf8). If your input is not utf8 encode you get a UnicodeDecodeError .

What can be done?

If you are not sure of the initial encoding and just want to leave everything as is you can just declare a Latin1 encoding. It just changes every byte in the unicode character having that code. The ensure_ascii is a bit different: it just allows any byte in the resulting json string, which may lead to non portable Json:

The RFC does not explicitly forbid JSON strings which contain byte sequences that don't correspond to valid Unicode characters (eg unpaired UTF-16 surrogates), but it does note that they may cause interoperability problems. By default, this module accepts and outputs (when present in the original str) code points for such sequences.

So this is a bullet proof way:

def convert2json(filename):
    with open(filename) as I:
        for line in I:
            d = {"data": line}
            print(json.dumps(d, encoding='Latin1'))

Simply a non ascii character in the input file ,say '\\x..' , will be coded in the Json as '\\u00..'

Answer 2

Well, to skip those lines you can use this:

#!/usr/bin/env python
import json

def convert2json(filename):
    with open(filename) as I:
        for line in I:
            try:
                d = {"data": line}
                print(json.dumps(d, ensure_ascii=False))
            except:
                continue

if __name__ == "__main__":
    import sys

    convert2json(sys.argv[1])  

I have wrapped the code in the loop in a try/except block.
This way when an error occurs, it will be muffled. You will see no output from the current line and the script will continue to the next one.

However, my tests didn't throw an error when tried with the provided part of the file. Could you tell us what the encoding of your file is? Are you sure that the problem is caused by those characters? Try adding a print() statement in your original code and a counter so you can identify the correct invalid line.