我可以检测字符串中使用的文本编解码器吗？

Question

I'm reading a string from a file(which anybody can modify) and don't know which type of coded the string is. Is there any function like

 getCodec = mystring.getCodec()

which return something like

 getCodec = 'utf-8' 

or

getCodec = 'ascii'

?

Answer 1

No, there is no such function, because files do not record what codec was used to write the text contained.

If there is more context (like a more specific format such as HTML or XML) then you can determine the codec because the standard specifies a default or allows for annotating the data with the codec, but otherwise you are reduced to guessing based on the contents (which is what tools like chardet do).

For a file that anyone can modify, you have no hope but to document clearly what codec should be used.

Answer 2

You can use a 3rd-party chardet module.

>>> import chardet
>>> chardet.detect(b'\xed\x95\x9c\xea\xb8\x80')  # u'한글'.encode('utf-8')
{'confidence': 0.7525, 'encoding': 'utf-8'}
>>> chardet.detect(b'\xc7\xd1\xb1\xdb')
{'confidence': 0.99, 'encoding': 'EUC-KR'}  # u'한글'.encode('euc-kr')

NOTE: chardet is not foolproof, and if a file is small enough can easily guess wrong.

Answer 3

If you cannot use chardet and have no chance of specifying the encoding in advance, I think your only remaining recourse is to simply guess at it. You could do something like this:

# Add whichever you want to the list, but only end it in a codec like latin1 that never fails
codecs = ["utf-8", "euc-kr", "shift-jis", "latin1"]

def try_decode(text):
    for codec in codecs:
        try:
            return text.decode(codec)
        except UnicodeError:
            continue