scikit-learn中TF-IDF向量的組特征

Question

我正在使用scikit-learn通過以下代碼來訓練基於TF-IDF特征向量的文本分類模型：

model = naive_bayes.MultinomialNB()
feature_vector_train = TfidfVectorizer().fit_transform(X)
model.fit(self.feature_vector_train, Y)

我需要對提取的特征按其TF-IDF權重的降序進行排序，並將它們分組為兩個不重疊的特征集，最后訓練兩個不同的分類模型。 如何將主要特征向量分組為奇數集和偶數集？

Answer 1

TfidfVectorizer的結果是一個nxm矩陣n是文檔數， m是唯一字數。 因此， feature_vector_train每一列都對應於數據集中的特定單詞。 通過改編本教程中的解決方案，應該可以提取最高和最低加權的單詞：

vectorizer = TfidfVectorizer()
feature_vector_train = vectorizer.fit_transform(X)
feature_names = vectorizer.get_feature_names()

total_tfidf_weights = feature_vector_train.sum(axis=0) #this assumes you only want a straight sum of each feature's weight across all documents
#alternatively, you could use vectorizer.transform(feature_names) to get the values of each feature in isolation

#sort the feature names and the tfidf weights together by zipping them
sorted_names_weights = sorted(zip(feature_names, total_tfidf_Weights), key = lambda x: x[1]), reversed=True) #the key argument tells sorted according to column 1. reversed means sort from largest to smallest
#unzip the names and weights
sorted_features_names, sorted_total_tfidf_weights = zip(*sorted_names_weights)

從這一點上，您應該能夠根據需要分離功能。 一旦將它們分為兩組，即group1和group2 ，就可以將它們分成兩個矩陣，如下所示：

#create a feature_name to column index mapping
column_mapping = dict((name, i) for i, name, in enumerate(feature_names))

#get the submatrices
group1_column_indexes = [column_mapping[feat] for feat in group1]
group1_feature_vector_train  = feature_vector_train[:,group1_column_indexes] #all rows, but only group1 columns

group2_column_indexes = [column_mapping[feat] for feat in group2]
group2_feature_vector_train  = feature_vector_train[:,group2_column_indexes]