如何在某些例外情况下将句子分解为单词

Question

I am working on a text classification project, and I need to split a sentence into words so I can calculate the probability of it being positive or negative. The problem is the word "not" , when ever it comes in, it changes the sentence which was suppose to be positive to negative, but my system still categorizes the sentence as positive which makes it wrong.

My idea is to find a way to split the sentence into words with an exception of 'not'

For example, " she is not beautiful "

Instead of getting "she", "is", "not", "beautiful"

I want to get "she", "is", "not beautiful "

Answer 1

You can use re.split with a negative lookbehind for the word "not"

import re
mystr = "she is not beautiful"
re.split("(?<!not)\s", mystr)
#['she', 'is', 'not beautiful']

The regular expression pattern is:

• (?<!not) : Negative lookbehind for "not"
• \\s : Any whitespace character

Answer 2

You can also try to

1. Split the text by 'not'

2. Take the first element in the new list and split it and add it to another list to be returned

3. for other elements of list from step1. we split each item and add not to the first item.

def my_seperator(text):
    text = text.strip()
    my_text = []
    text = text.split('not')
    my_text = my_text + text[0].split()
    for t in text[1:]:
        temp_text = t.split()
        my_text.append('not '+temp_text[0])
        my_text = my_text+temp_text[1:]
    return my_text

>>> my_seperator('she is not beautiful . but not that she is ugly. Maybe she is not my type')
['she', 'is', 'not beautiful', '.', 'but', 'not that', 'she', 'is', 'ugly.', 'Maybe', 'she', 'is', 'not my', 'type']

Although like @pault mentioned regular expression is the way to go.