python中的列表和带标记的句子，用引号“'”分隔，带空格和不带空格

Question

I have a dataset, and by regex I extracted the data. I used the sent_tokenize method of NLTK to define for me the sentence boundary.

tok = sent_tokenize(str(all_text))
print(tok[0])
It give me this output:


# List of string 
tok = ['Hi ' ,  hello at 'this ', there 'from ']

Now the annotated data that I have extracted from this dataset looks like:

i = ['there' , 'hello', 'Hi']

If you see, in the tok list the first quotation is with word and the closing quotation is with a space. But in the i list, an element of a list is closed with quotation without space. When I want to check if any element of i in tok , it should give me a result. but can not detect the text inside tok .

Answer 1

这应该可以解决您的问题：

tok = [j.strip() for j in tok]

Answer 2

Im not sure why sent_tokenize tokenizes each word in the sentence for you. but if you want tokens for each sentence, try something like this..

from nltk import PunktSentenceTokenizer, word_tokenize
tokenizer = nltk.tokenize.punkt.PunktSentenceTokenizer()
tokens = [word_tokenize(i) for i in tokenizer.tokenize(all_text)]