我如何在python 3.6中转换字符？

Question

I'm a little confused on how to unescape characters in python. I am parsing some HTML using BeautifulSoup, and when I retrieve the text content it looks like this:

\u00a0\n\n\n\r\nState-of-the-art security and 100% uptime SLA.\u00a0\r\n\n\n\r\nOutstanding support

I'd like for it to look like this:

State-of-the-art security and 100% uptime SLA. Outstanding support

Here is my code below:

    self.__page = requests.get(url)
    self.__soup = BeautifulSoup(self.__page.content, "lxml")
    self.__page_cleaned = self.__removeTags(self.__page.content) #remove script and style tags
    self.__tree = html.fromstring(self.__page_cleaned) #contains the page html in a tree structure
    page_data = {}
    page_data["content"] =  self.__tree.text_content()

How do I remove those encoded backslashed characters? I've looked everywhere and nothing has worked for me.

Answer 1

You can convert those escape sequences to proper text using the codecs module.

import codecs

s = r'\u00a0\n\n\n\r\nState-of-the-art security and 100% uptime SLA.\u00a0\r\n\n\n\r\nOutstanding support'

# Convert the escape sequences
z = codecs.decode(s, 'unicode-escape')
print(z)
print('- ' * 20)

# Remove the extra whitespace
print(' '.join(z.split()))       

output

    [several blank lines here]
 



State-of-the-art security and 100% uptime SLA. 



Outstanding support
- - - - - - - - - - - - - - - - - - - - 
State-of-the-art security and 100% uptime SLA. Outstanding support

The codecs.decode(s, 'unicode-escape') function is quite versatile. It can handle simple backslash escapes, like those newline and carriage return sequences ( \\n and \\r ), but its main strength is handling Unicode escape sequences, like the \  , which is just a nonbreak space char. But if your data had other Unicode escapes in it, like those for foreign alphabetic chars or emojis, it would handle them too.

---

As Evpok mentions in a comment, this won't work if the text string contains actual Unicode characters as well as Unicode \\u\u003c/code> or \\U escape sequences.

From the codecs docs :

unicode_escape

Encoding suitable as the contents of a Unicode literal in ASCII-encoded Python source code, except that quotes are not escaped. Decodes from Latin-1 source code. Beware that Python source code actually uses UTF-8 by default.

Also see the docs for codecs.decode .

Answer 2

You could use regular expressions:

import re

s = '\u00a0\n\n\n\r\nState-of-the-art security and 100% uptime SLA.\u00a0\r\n\n\n\r\nOutstanding support'
s = ' '.join(re.findall(r"[\w%\-.']+", s))

print(s) #output: State-of-the-art security and 100% uptime SLA. Outstanding support

re.findall("exp", s) returns a list of all substrings of s which match the pattern "exp". In the case of "[\\w]+" all combinations of letters or numbers (no hex string like "\ "):

['State', 'of', 'the', 'art', 'security', 'and', '100', 'uptime', 'SLA', 'Outstanding', 'support'] 

You can include characters by adding them to the expression like so:

re.findall(r"[\w%.-']+", s)    # added "%", "." and "-" ("-"needs to be escaped by "\")

' '.join(s) returns a string of all elements seperated by the string in the quotes (in this case a space).