查找并拆分单词后面的某些字符

Question

I'm trying to use regular expressions to split text on punctuation, only when the punctuation follows a word and proceeds a space or the end of the string.

I've tried ([a-zA-Z])([,;.-])(\\s|$)

But when I want to split in Python, it includes the last character of the word.

I want to split it like this:

text = 'Mr.Smith is a professor at Harvard, and is a great guy.'
splits = ['Mr.Smith', 'is', 'a', 'professor', 'at', 'Harvard', ',', 'and', 'a', 'great', 'guy', '.']

Any help would be greatly appreciated!

Answer 1

It seems you want to do tokenize. Try nltk

http://text-processing.com/demo/tokenize/

from nltk.tokenize import TreebankWordTokenizer
splits = TreebankWordTokenizer().tokenize(text)

Answer 2

You may use

re.findall(r'\w+(?:\.\w+)*|[^\w\s]', s)

See the regex demo .

Details

• \\w+(?:\\.\\w+)* - 1+ word chars followed with 0 or more occurrences of a dot followed with 1+ word chars
• | - or
• [^\\w\\s] - any char other than a word and whitespace char.

Python demo :

import re
rx = r"\w+(?:\.\w+)*|[^\w\s]"
s = "Mr.Smith is a professor at Harvard, and is a great guy."
print(re.findall(rx, s))

Output: ['Mr.Smith', 'is', 'a', 'professor', 'at', 'Harvard', ',', 'and', 'is', 'a', 'great', 'guy', '.'] .

This approach can be further precised. Eg tokenizing only letter words, numbers and underscores as punctuation:

re.findall(r'[+-]?\d*\.?\d+|[^\W\d_]+(?:\.[^\W\d_]+)*|[^\w\s]|_', s)

See the regex demo

Answer 3

You can first split on ([.,](?=\\s)|\\s) and then filter out empty or blanks strings:

In [16]: filter(lambda s: not re.match(r'\s*$', s) , re.split(r'([.,](?=\s)|\s)',  'Mr.Smith is a professor at Har
    ...: vard, and is a great guy.'))
Out[16]: 
['Mr.Smith',
 'is',
 'a',
 'professor',
 'at',
 'Harvard',
 ',',
 'and',
 'is',
 'a',
 'great',
 'guy.']