Python 3：将 UTF-8 unicode 印地语文字转换为 Unicode

Question

I have a string of UTF-8 literals

'\\xe0\\xa4\\xb9\\xe0\\xa5\\x80 \\xe0\\xa4\\xac\\xe0\\xa5\\x8b\\xe0\\xa4\\xb2' which covnverts to

ही बोल in Hindi. I am unable convert string a to bytes

a = '\xe0\xa4\xb9\xe0\xa5\x80 \xe0\xa4\xac\xe0\xa5\x8b\xe0\xa4\xb2'
#convert a to bytes
#also tried a = bytes(a,'utf-8')
a = a.encode('utf-8')
s = str(a,'utf-8')

The string is converted to bytes but contains wrong unicode literals

RESULT : b'\\xc3\\xa0\\xc2\\xa4\\xc2\\xb9\\xc3\\xa0\\xc2\\xa5\\xc2\\x80 \\xc3\\xa0\\xc2\\xa4\\xc2\\xac\\xc3\\xa0\\xc2\\xa5\\xc2\\x8b\\xc3\\xa0\\xc2\\xa4\\xc2\\xb2' which prints - हॠबà¥à¤²

EXPECTED : It should be b'\\xe0\\xa4\\xb9\\xe0\\xa5\\x80\\xe0\\xa4\\xac\\xe0\\xa5\\x8b\\xe0\\xa4\\xb2 which will be ही बोल

Answer 1

Use the raw-unicode-escape codec to encode the string as bytes, then you can decode as UTF-8.

>>> s = '\xe0\xa4\xb9\xe0\xa5\x80 \xe0\xa4\xac\xe0\xa5\x8b\xe0\xa4\xb2'
>>> s.encode('raw-unicode-escape').decode('utf-8')
'ही बोल'

This is something of a workaround; the ideal solution would be to prevent the source of the data stringifying the original bytes.

Answer 2

Your original string was likely decoded as latin1 . Decode it as UTF-8 instead if possible, but if received messed up you can reverse it by encoding as latin1 again and decoding correctly as UTF-8:

>>> s = '\xe0\xa4\xb9\xe0\xa5\x80 \xe0\xa4\xac\xe0\xa5\x8b\xe0\xa4\xb2'
>>> s.encode('latin1').decode('utf8')
'ही बोल'

Note that latin1 encoding matches the first 256 Unicode code points, so U+00E0 ( '\\xe0' in a Python 3 str object) becomes byte E0h ( b'\\xe0' in a Python 3 bytes object). It's a 1:1 mapping between U+0000-U+00FF and bytes 00h-FFh.