定制的非ASCII字符标记

Question

I've looked around for a custom-made solution, but I couldn't find a solution for a use case that I am facing.

Use Case

I'm building a 'website' QA test where the script will go through a bulk of HTML documents, and identify any rogue characters. I cannot use pure non-ascii method since the HTML documents contain characters such as ">" and other minor characters. Therefore, I am building up a unicode rainbow dictionary that identifies some of the common non-ascii characters that my team and I frequently see. The following is my Python code.

# -*- coding: utf-8 -*-

import re

unicode_rainbow_dictionary = {
    u'\u00A0':' ',
    u'\uFB01':'fi',
}

strings = ["This contains the annoying non-breaking space","This is fine!","This is not fine!"]

for string in strings:
    for regex in unicode_rainbow_dictionary:
        result = re.search(regex,string)
        if result:
            print "Epic fail! There is a rogue character in '"+string+"'"
        else:
            print string

The issue here is that the last string in the strings array contains a non-ascii ligature character (the combined fi). When I run this script, it doesn't capture the ligature character, but it captures the non-breakable space character in the first case.

What is leading to the false positive?

Answer 1

Use Unicode strings for all text as @jgfoot points out. The easiest way to do this is to use from __future__ to default to Unicode literals for strings. Additionally, using print as a function will make the code Python 2/3 compatible:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals,print_function
import re

unicode_rainbow_dictionary = {
    '\u00A0':' ',
    '\uFB01':'fi',
}

strings = ["This contains the\xa0annoying non-breaking space","This is fine!","This is not fine!"]

for string in strings:
    for regex in unicode_rainbow_dictionary:
        result = re.search(regex,string)
        if result:
            print("Epic fail! There is a rogue character in '"+string+"'")
        else:
            print(string)

Answer 2

If you have the possibility then switch to Python 3 as soon as possible! Python 2 is not good at handling unicode whereas Python 3 does it natively.

for string in strings:
    for character in unicode_rainbow_dictionary:
        if character in string:
            print("Rogue character '" + character + "' in '" + string + "'")

I couldn't get the non-breaking space to occur in my test. I got around that by using "This contains the annoying" + chr(160) + "non-breaking space" after which it matched.

Answer 3

Your code doesn't work as expected because, in your "strings" variable, you have unicode characters in non-unicode strings. You forgot to put the "u" in front of them to signal that they should be treated as unicode strings. So, when you search for a unicode string inside a non-unicode string, it doesn't work as expected

If you change this to:

strings = [u"This contains the annoying non-breaking space",u"This is fine!",u"This is not fine!"]

It works as expected.

Solving unicode headaches like this is a major benefit of Python 3.

Here's an alternative approach to your problem. How about just trying to encode the string as ASCII, and catching errors if it doesn't work?:

def is_this_ascii(s):
    try:
        ignore = unicode(s).encode("ascii")
        return True
    except (UnicodeEncodeError, UnicodeDecodeError):
        return False

strings = [u"This contains the annoying non-breaking space",u"This is fine!",u"This is not fine!"]

for s in strings:
    print(repr(is_this_ascii(s)))

##False
##True
##False