在一個列表中對我的CSV進行標記，而不是使用Python進行分離

Question

我想在一個列表而不是單獨的列表中將我的CSV標記化？

with open ('train.csv') as file_object:
    for trainline in file_object:
        tokens_train = sent_tokenize(trainline)
        print(tokens_train)

這是我得到輸出的方式：

['2.1 Separated of trains']
['Principle: The method to make the signal is different.']
['2.2 Context']

我希望所有這些都在一個列表中

['2.1 Separated of trains','Principle: The method to make the signal is different.','2.2 Context']

Answer 1

由於sent_tokenize()返回一個列表，因此您每次只需擴展一個起始列表。

alltokens = []

with open ('train.csv') as file_object:
    for trainline in file_object:
        tokens_train = sent_tokenize(trainline)
        alltokens.extend(tokens_train)
    print(alltokens)

或者列表理解：

with open ('train.csv') as file_object:
    alltokens = [token for trainline in file_object for token in sent_tokenize(trainline)]
print(alltokens)

即使sent_tokenize()返回長於1的列表，這兩種解決方案都將起作用。

Answer 2

初始化一個空列表

out = []

在循環內部附加項目。

out.append(tokens_train)

也許你必須修改你的標記器。