在一个列表中对我的CSV进行标记，而不是使用Python进行分离

Question

我想在一个列表而不是单独的列表中将我的CSV标记化？

with open ('train.csv') as file_object:
    for trainline in file_object:
        tokens_train = sent_tokenize(trainline)
        print(tokens_train)

这是我得到输出的方式：

['2.1 Separated of trains']
['Principle: The method to make the signal is different.']
['2.2 Context']

我希望所有这些都在一个列表中

['2.1 Separated of trains','Principle: The method to make the signal is different.','2.2 Context']

Answer 1

由于sent_tokenize()返回一个列表，因此您每次只需扩展一个起始列表。

alltokens = []

with open ('train.csv') as file_object:
    for trainline in file_object:
        tokens_train = sent_tokenize(trainline)
        alltokens.extend(tokens_train)
    print(alltokens)

或者列表理解：

with open ('train.csv') as file_object:
    alltokens = [token for trainline in file_object for token in sent_tokenize(trainline)]
print(alltokens)

即使sent_tokenize()返回长于1的列表，这两种解决方案都将起作用。

Answer 2

初始化一个空列表

out = []

在循环内部附加项目。

out.append(tokens_train)

也许你必须修改你的标记器。