查找具有指定 tf-idf 分数的单词

Question

如何从所有单词中获得 tf-idf 分数最低的单词？

tfidf_vect = TfidfVectorizer(analyzer=clean)
print(tfidf_vect.fit_transform(df['text']))

[output]
(0, 11046)  0.1907144678156909
(0, 4791)   0.3125060892887963
(0, 7026)   0.15156899671911586
(0, 1534)   0.3125060892887963
...

我想用它们的多个索引说出分数低于 0.1 的单词。 我知道我正在使用 csr_matrix 并将其转换为数组以更轻松地对其进行处理，但无法使其正常工作。

Answer 1

我想到的最简单的方法是使用简单的numpy功能进行过滤，然后在必要时将其转换为sparse matrix 。

corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
threshold = 0.5
df = pd.DataFrame({'text':corpus})
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(df['text'])
rows, columns = np.where(X.toarray() > threshold)
print(f'Rows: {list(rows)}\nColumns: {list(columns)}')
filtered_sparse_matrix = csr_matrix((X.toarray()[X.toarray()>threshold], (rows, columns)), X.shape)
print(f'Final Matrix:\n{filtered_sparse_matrix}')

输出：

Rows: [0, 1, 1, 2, 2, 2, 3]
Columns: [2, 1, 5, 0, 4, 7, 2]
Final Matrix:
  (0, 2)    0.5802858236844359
  (1, 1)    0.6876235979836938
  (1, 5)    0.5386476208856763
  (2, 0)    0.511848512707169
  (2, 4)    0.511848512707169
  (2, 7)    0.511848512707169
  (3, 2)    0.5802858236844359