处理字符串中的十进制转义

Question

I have a file of strings one per line in which non-ascii characters have been escaped with decimal code points. One example line is:

mj\\\\195\\\\164ger

(The double backslashes are in the file exactly as printed)

I would like to process this string to produce

mjäger

. Conventionally, python uses hexadecimal escapes rather than decimal escapes (eg, the above string would be written as mj\\xc3\\xa4ger , which python can decode:

>>> by=b'mj\xc3\xa4ger'
>>> by.decode('utf-8')
'mjäger'

Python, however, doesn't recognize the decimal escape right away.

I have written a method that correctly manipulates the strings to produce hexadecimal escapes, but these escapes are themselves escaped. How can I get python to process these hexadecimal escapes to create the final string?

import re

hexconst=["0","1","2","3","4","5","6","7","8","9","a","b","c","d","e","f"]
escapes=re.compile(r"\\[0-9]{3}")
def dec2hex(matchobj):
    dec=matchobj.group(0)
    dec=int(dec[1:])
    digit1=dec//16 #integer division
    digit2=dec%16 
    hex="\\x" + hexconst[digit1] + hexconst[digit2]
    return hex

line=r'mj\195\164ger'
print(escapes.sub(dec2hex,line)) #Outputs mj\xc3\xa4ger

What is the final step I'm missing to convert the output of the above from mj\\xc3\\xa4ger to mjäger ? Thanks!

Answer 1

It's much easier. re.sub() can take a callback function instead of a replacement string as an argument:

>>> import re
>>> line=r'mj\195\164ger'
>>> def replace(match):
...     return chr(int(match.group(1)))

>>> regex = re.compile(r"\\(\d{1,3})")
>>> new = regex.sub(replace, line)
>>> new
'mj\xc3\xa4ger'
>>> print new
mjäger

In Python 3, strings are Unicode strings, so if you're working with encoded input (like UTF-8 encoded content), then you need to use the proper type which is bytes :

>>> line = rb'mj\195\164ger'
>>> regex = re.compile(rb"\\(\d{1,3})")
>>> def replace(match):
...     return int(match.group(1)).to_bytes(1, byteorder="big")

>>> new = regex.sub(replace, line)
>>> new
b'mj\xc3\xa4ger'
>>> print(new.decode("utf-8"))
mjäger