使用python将句子列表拆分为单词列表

Question

I have a list of sentence

s: 'hello everyone', 'how are you',..., 'i am fine'.

I would like to split this list of sentences to a list of words.

So my expected result:

[['hello', 'everyone'], ['how', 'are', 'you'], .., ['i', 'am', 'fine]] 

I try like this :

def split_list(sentence):
    for s in sentence:
        s=s.split()
    return s

but i got one list of words, not a list of list of words.

['hello', 'everyone', 'how', 'are', 'you', .., 'i', 'am', 'fine]

Answer 1

It's not very clear on what sentence refers to in your function split_list , but if it is a list of strings like ['hello everyone', 'how are you', 'i am fine'] , you end up overwriting the same string s on every iteration, and end up getting the result of the last iteration, ie ['i', 'am', 'fine']

So you need to ensure that you are collecting all your results in a list of lists and returning that.

You can do that list-comprehension like so, assuming it is a list of strings like above

s = ['hello everyone', 'how are you', 'i am fine']

def split_list(sentence):
    # Split each sentence in the list, and append to result list
    return [item.split() for item in sentence]

print(split_list(s))

Or a normal for loop

s = ['hello everyone', 'how are you', 'i am fine']

def split_list(sentence):
    result = []
    #Split each sentence in the list, and append to result list
    for s in sentence:
        result.append(s.split())
    return result

print(split_list(s))

The output will be same for both cases.

[['hello', 'everyone'], ['how', 'are', 'you'], ['i', 'am', 'fine']]

Answer 2

This can just be done with a list comprehension.

s = ['hello everyone', 'how are you', 'i am fine']
s2 = [c.split() for c in s]
print(s2) # [['hello', 'everyone'], ['how', 'are', 'you'], ['i', 'am', 'fine']]

Answer 3

You have to save the result of each iteration in a list by initializing an empty list before the loop and appending each result in the loop:

def split_list(sentence):
    L = []
    for s in sentence:
        L.append(s.split())
    return L

Otherwise the function will return only the result of the last iteration.

Answer 4

from nltk import word_tokenize
s = ['hello everyone', 'how are you', 'i am fine']

token = [word_tokenize(x) for x in s]
print(token)

o/p
[['hello', 'everyone'], ['how', 'are', 'you'], ['i', 'am', 'fine']]