Python正则表达式：删除下划线和破折号，除非它们在字符串替换指令中

Question

I'm pre-processing a string. I have a dictionary of 10k string substitutions (eg "John Lennon": "john_lennon" ). I want to replace all other punctuation with a space.

The problem is some of these string substitutions contain underscores or hyphens, so I want to replace punctuation (except full stops) with spaces unless the word is contained in the keys of this dict. I also want to do it in one Regex expression since the text corpus is quite large and this could be a bottleneck.

So far, I have:

import re
input_str = "John Lennon: a musician, artist and activist."
multi_words = dict((re.escape(k), v) for k, v in multi_words.items())
pattern = re.compile("|".join(multi_words.keys()))
output_str = pattern.sub(lambda m: multi_words[re.escape(m.group(0))], input_str)

This replaces all strings using the keys in a dict. Now I just need to also remove punctuation in the same pass. This should return "john_lennon a musician artist and activist."

Answer 1

You could handle the punctuation you would like to remove like the entries in your dictionary:

pattern = re.compile("|".join(multi_words.keys()) + r'|_|-')

and

multiwords['_'] = ' '
multiwords['-'] = ' '

Then these occurrences are treated like your key words.

But let me remind you that your code only works for a certain set of regular expressions. If you have the pattern foo.*bar in your keys and that matches a string like foo123bar , you will not find the corresponding value to the key by passing foo123bar through re.escape() and then searching in your multiword dictionary for it.

I think the whole escaping you do should be removed and the code should be commented to make clear that only fixed strings are allowed as keys, not complex regular expressions matching variable inputs.

Answer 2

You can add punctuations (excluding full stop) in a character set as part of the items to match, and then handle punctuations and dict keys separately in the substitution function:

import re
import string

punctuation = string.punctuation.replace('.', '')
pattern = re.compile("|".join(multi_words.keys())+
                     "|[{}]".format(re.escape(punctuation)))


def func(m):
   m = m.group(0)
   print(m, re.escape(m))
   if m in string.punctuation:
      return ''
   return multi_words[re.escape(m)]

output_str = pattern.sub(func , input_str)
print(output_str)
# john_lennon a musician artist and activist. 

Answer 3

You could do it by adding one more alternative to the constructed regex which matches a single punctuation character. When the match is processed, a match not in the dictionary can be replaced with a space, using dictionary's get method. Here, I use [,:;_-] but you probably want to replace other characters.

Note: I moved the call to re.escape into the construction of the regex to avoid having to call it on every match.

import re
input_str = "John Lennon: a musician, artist and activist."
pattern = re.compile(("|".join(map(re.escape, multi_words.keys())) + "|[,:;_-]+")
output_str = pattern.sub(lambda m: multi_words.get(m.group(0), ' '), input_str)

Answer 4

You may use a regex like (?:alt1|alt2...|altN)|([^\\w\\s.]+) and check if Group 1 (that is, any punctuation other than . ) was matched. If yes, replace with an empty string:

pattern = re.compile(r"(?:{})|([^\w\s.]+)".format("|".join(multi_words.keys())))
output_str = pattern.sub(lambda m: "" if m.group(1) else multi_words[re.escape(m.group(0))], input_str)

See the Python demo .

A note about _ : if you need to remove it as well, use r"(?:{})|([^\\w\\s.]+|_+)" because [^\\w\\s.] (matching any char other than word, whitespace and . chars) does not match an underscore (a word char) and you need to add it as a separate alternative.

Note on Unicode: if you deal with Unicode strings, in Python 2.x, pass re.U or re.UNICODE modifier flag to the re.compile() method.