How to remove all the escape sequences from a list of strings?

Question

I want to remove all types of escape sequences from a list of strings. How can I do this? input:

['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']

output:

['william', 'short', 'twitter', 'video', 'guy', 'ray']

http://docs.python.org/reference/lexical_analysis.html#string-literals

Answer 1

If you want to strip out some characters you don't like, you can use the translate function to strip them out:

>>> s="\x01\x02\x10\x13\x20\x21hello world"
>>> print(s)
 !hello world
>>> s
'\x01\x02\x10\x13 !hello world'
>>> escapes = ''.join([chr(char) for char in range(1, 32)])
>>> t = s.translate(None, escapes)
>>> t
' !hello world'

This will strip out all these control characters:

   001   1     01    SOH (start of heading)
   002   2     02    STX (start of text)
   003   3     03    ETX (end of text)
   004   4     04    EOT (end of transmission)
   005   5     05    ENQ (enquiry)
   006   6     06    ACK (acknowledge)
   007   7     07    BEL '\a' (bell)
   010   8     08    BS  '\b' (backspace)
   011   9     09    HT  '\t' (horizontal tab)
   012   10    0A    LF  '\n' (new line)
   013   11    0B    VT  '\v' (vertical tab)
   014   12    0C    FF  '\f' (form feed)
   015   13    0D    CR  '\r' (carriage ret)
   016   14    0E    SO  (shift out)
   017   15    0F    SI  (shift in)
   020   16    10    DLE (data link escape)
   021   17    11    DC1 (device control 1)
   022   18    12    DC2 (device control 2)
   023   19    13    DC3 (device control 3)
   024   20    14    DC4 (device control 4)
   025   21    15    NAK (negative ack.)
   026   22    16    SYN (synchronous idle)
   027   23    17    ETB (end of trans. blk)
   030   24    18    CAN (cancel)
   031   25    19    EM  (end of medium)
   032   26    1A    SUB (substitute)
   033   27    1B    ESC (escape)
   034   28    1C    FS  (file separator)
   035   29    1D    GS  (group separator)
   036   30    1E    RS  (record separator)
   037   31    1F    US  (unit separator)

For Python newer than 3.1, the sequence is different:

>>> s="\x01\x02\x10\x13\x20\x21hello world"
>>> print(s)
 !hello world
>>> s
'\x01\x02\x10\x13 !hello world'
>>> escapes = ''.join([chr(char) for char in range(1, 32)])
>>> translator = str.maketrans('', '', escapes)
>>> t = s.translate(translator)
>>> t
' !hello world'

Answer 2

Something like this?

>>> from ast import literal_eval
>>> s = r'Hello,\nworld!'
>>> print(literal_eval("'%s'" % s))
Hello,
world!

Edit : ok, that's not what you want. What you want can't be done in general, because, as @Sven Marnach explained, strings don't actually contain escape sequences. Those are just notation in string literals.

You can filter all strings with non-ASCII characters from your list with

def is_ascii(s):
    try:
        s.decode('ascii')
        return True
    except UnicodeDecodeError:
        return False

[s for s in ['william', 'short', '\x80', 'twitter', '\xaa',
             '\xe2', 'video', 'guy', 'ray']
 if is_ascii(s)]

Answer 3

You could filter out "words" that are not alphanumeric using a list comprehension and str.isalnum() :

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']
>>> [word for word in l if word.isalnum()]
['william', 'short', 'twitter', 'video', 'guy', 'ray']

If you wish to filter out numbers, too, use str.isalpha() instead:

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray', '456']
>>> [word for word in l if word.isalpha()]
['william', 'short', 'twitter', 'video', 'guy', 'ray']

Answer 4

This cannot be done, at least at the broad scope you are asking. As others have mentioned, runtime python doesn't know the difference between the something with escape sequences, and something without.

Example:

print ('\x61' == 'a')

prints True . So there's no way to find the difference between these two strings, unless you try some static analysis of your python script.

Answer 5

I had similar issues while converting from hexadimal to String.This is what finally worked in python Example

list_l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']
decode_data=[]
for l in list_l:
    data =l.decode('ascii', 'ignore')
    if data != "":
        decode_data.append(data)

# output :[u'william', u'short', u'twitter', u'video', u'guy', u'ray']

Answer 6

The following worked for me in Python 3.6:

word = '\t\t\t\t\t\ntest\t msg'
filter = ''.join([chr(i) for i in range(1, 32)])
word.translate(str.maketrans('', '', filter))

Output:

'test msg'

Source here