如何检查是否已成功在utf-8中进行编码

Question

Given a string

u ='abc'

which syntax is the right one to encode into utf8?

u.encode('utf-8')

or

u.encode('utf8')

And how do I know that I have already encoded in utr-8?

Answer 1

First of all you need to make a distinction if you're talking about Python 2 or Python 3 because unicode handling is one of the biggest differences between the two versions.

Python 2

• unicode type contains text characters
• str contains sequences of 8-bit bytes, sometimes representing text in some unspecified encoding
• s.decode(encoding) takes a sequence bytes and builds a text string out of it, once given the encoding used by the bytes. It goes from str to unicode , for example "Citt\\xe0".decode("iso8859-1") will give you the text "Città" (Italian for city) and the same will happen for "Citt\\xc3\\xa0".decode("utf-8") . The encoding may be omitted and in that case the meaning is "use the default encoding".
• u.encode(encoding) takes a text string and builds the byte sequence representing it in the given encoding, thus reversing the processing of decode . It goes from unicode to str . As above the encoding can be omitted.

Part of the confusion when handling unicode with Python is that the language tries to be a bit too smart and does things automatically.

For example you can call encode also on an str object and the meaning is "encode the text that comes from decoding these bytes when using the default encoding, eventually using the specified encoding or the default encoding if not specified".

Similarly you can also call decode on an unicode object, meaning "decode the bytes that come from this text when using the default encoding, eventually using the specified encoding".

For example if I write

u"Citt\u00e0".decode("utf-8")

Python gives as error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\\xe0' in position 3: ordinal not in range(128)

NOTE: the error is about encoding that failed, while I asked for decoding . The reason is that I asked to decode text (nonsense because that is already "decoded"... it's text) and Python decided to first encode it using the "ascii" encoding and that failed. IMO much better would have to just not have decode on unicode objects and not have encode on string objects: the error message would have been clearer.

More confusion is that in Python 2 str is used for unencoded bytes , but it's also used everywhere for text and for example string literals are str objects.

Python 3

To solve some of the issues Python 3 made a few key changes

• str is for text and contains unicode characters, string literals are unicode text
• unicode type doesn't exist any more
• bytes type is used for 8-bit bytes sequences that may represent text in some unspecified encoding

For example in Python 3

'Città'.encode('iso8859-1') → b'Citt\xe0'
'Città'.encode('utf-8')     → b'Citt\xc3\xa0'

also you cannot call decode on text strings and you cannot call encode on byte sequences.

Failures

Sometimes encoding text into bytes may fail, because the specified encoding cannot handle all of unicode. For example iso8859-1 cannot handle Chinese. These errors can be processed in a few ways like raising an exception (default), or replacing characters that cannot be encoded with something else.

The encoding utf-8 however is able to encode any unicode character and thus encoding to utf-8 never fails.

Also decoding may fail, because the sequence of bytes may make no sense in the specified encoding. For example the sequence of bytes 0x43 0x69 0x74 0x74 0xE0 cannot be interpreted as utf-8 because the byte 0xE0 cannot appear without a proper prefix.

There are encodings like iso8859-1 where however decoding cannot fail because any byte 0..255 has a meaning as a character. Most "local encodings" are of this type... they map all 256 possible 8-bit values to some character, but only covering a tiny fraction of the unicode characters.

Decoding using iso8859-1 will never raise an error (any byte sequence is valid) but of course it can give you nonsense text if the bytes where using another encoding.

Answer 2

First solution:

isinstance(u, unicode)

Second solution:

try:
    u.decode('utf-8')
    print "string is UTF-8, length %d bytes" % len(string)
except UnicodeError:
    print "string is not UTF-8"