将文本转换为单词表时，如何保留法语特殊字符？

Question

I am using a snippet of code for a markov chain sentence generator. Works fine in English, but in French, it doesn't print out the special characters (é, è, etc.).

This is the part that reads a file and creates a wordlist from it. I use the print statements as controls, which allow me to see that print(text) prints the special characters, but once the word is added to the wordlist, they disappear.

def wordlist(filename):
    f = open(filename, mode='r')
    text = f.read()
    print(text)
    wordlist = [fixCaps(w) for w in re.findall(r"[\w']+|[.,!?;]", text)]
    print(wordlist)
    f.close()
    return wordlist

How can I preserve the special characters when creating the word list? (I am running this on Windows 7 with Python 2.x)

Example of output:

Permettez-moi d'inscrire votre nom en tête de ce livre et au-
dessus même de sa dédicace; car c'est à vous, surtout, que j'en
dois la publication. En passant par votre magnifique plaidoirie,
mon oeuvre a acquis pour moi-même comme une autorité imprévue.
Acceptez donc ici l'hommage de ma gratitude, qui, si grande
qu'elle puisse être, ne sera jamais à la hauteur de votre
éloquence et de votre dévouement.
['Permettez', 'moi', "d'inscrire", 'votre', 'nom', 'en', 't', 'te', 'de', 'ce', 'livre', 'et', 'au', 'dessus', 'm', 'me', 'de', 'sa', 'd', 'dicace', ';', 'car', "c'est", 'vous', ',', 'surtout', ',', 'que', "j'en", 'dois', 'la', 'publication', '.', 'En', 'passant', 'par', 'votre', 'magnifique', 'plaidoirie', ',', 'mon', 'oeuvre', 'a', 'acquis', 'pour', 'moi', 'm', 'me', 'comme', 'une', 'autorit', 'impr', 'vue', '.', 'Acceptez', 'donc', 'ici', "l'hommage", 'de', 'ma', 'gratitude', ',', 'qui', ',', 'si', 'grande', "qu'elle", 'puisse', 'tre', ',', 'ne', 'sera', 'jamais', 'la', 'hauteur', 'de', 'votre', 'loquence', 'et', 'de', 'votre', 'd', 'vouement', '.']
En passant par votre magnifique plaidoirie, mon oeuvre a acquis pour moi m me comme une autorit impr vue.

Thanks

Answer 1

The words do not actually disappear, they are just not matched by your expression:

wordlist = [fixCaps(w) for w in re.findall(r"[\w']+|[.,!?;]", text)]

The escape \\w matches "word characters", but the interpretation of what actually a 'word character' is, varies per GREP implementation:

\\w stands for "word character". It always matches the ASCII characters [A-Za-z0-9_] . Notice the inclusion of the underscore and digits. In most flavors that support Unicode, \\w includes many characters from other scripts. There is a lot of inconsistency about which characters are actually included.
( https://www.regular-expressions.info/shorthand.html )

By default, Python 2.7's \\w matches only the basic limited set, but you can add flags to ask for more:

\\w
When the LOCALE and UNICODE flags are not specified, matches any alphanumeric character and the underscore; this is equivalent to the set [a-zA-Z0-9_] . With LOCALE , it will match the set [0-9_] plus whatever characters are defined as alphanumeric for the current locale. If UNICODE is set, this will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database.
( https://docs.python.org/2/library/re.html )

This suggests the following code (slightly adjusted to not use a file; it's the regex that makes the difference):

def wordlist(text):
    regex = re.compile (r"[\w']+|[.,!?;]", re.UNICODE)
    print(text)
    wordlist = [fixCaps(w) for w in re.findall(regex, text)]
    return wordlist

and indeed now accented characters get included:

['Permettez', 'moi', "d'inscrire", 'votre', 'nom', 'en', 't\xc3\xaate', 'de', 'ce', 'livre',
 'et', 'au', 'dessus', 'm\xc3\xaame', 'de', 'sa', 'd\xc3', 'dicace', ';', 'car', "c'est",
...
# etc.

Python 3 and onwards have better Unicode support, so you don't have to use that flag and your original code works as expected with \\w :

['Permettez', 'moi', "d'inscrire", 'votre', 'nom', 'en', 'tête', 'de',
 'ce', 'livre', 'et', 'au', 'dessus', 'même', 'de', 'sa', 'dédicace', ';',
   ...

(where an added bonus is that accented characters do not get output as \\x nn ).