使用前面带有非字母字符的空格拆分文本

Question

since I couldn't find any solution in the internet, I just thought of asking my question on here.

I want to split a given text at every punctuation. So not only after every sentence, but also after a comma for example. I came across the natural language toolkit (tltk) and regular expressions so far, but I had no success with them.

This is what works quite good, but does not fulfil my expectations completely:

sample_text = """With this example I wanna make the point clear... I hope you get it! There are many coding
languages out there, but which is the best? I would say there's no best. Change my mind - if you can!"""

split_text = nltk.tokenize.sent_tokenize(sample_text)
print(split_text)

#Output: ['With this example I wanna make the point clear...', 'I hope you get it!', 'There are many coding languages out there, but which is the best?', "I would say there's no best.", 'Change my mind - if you can!']

This is quite okay already, but I preferably would like to receive an output, which even splits the text at commas or a hyphen. So the output would look like this:

[
 'With this example I wanna make the point clear...',
 'I hope you get it!',
 'There are many coding languages out there,',
 'but which is the best?',
 "I would say there's no best.",
 'Change my mind -',
 'if you can!'
]

It's probably better to use regular expressions isn't it? But somehow I don't get it working. Thanks in advance, appreciate any help!

Answer 1

正则表达式效果很好，尝试在 .split() 中使用这个表达式

Answer 2

You could split the string on a space which is not preceded by a letter:

split_text = re.split('(?<=[^a-z]) ', sample_text, 0, re.I)
print(split_text)

Output:

[
 'With this example I wanna make the point clear...',
 'I hope you get it!',
 'There are many coding languages out there,',
 'but which is the best?',
 "I would say there's no best.",
 'Change my mind -',
 'if you can!'
]