Vectorizer fit_transform 按文档计数

Question

I have a scipy.sparse.csr.csr_matrix with shape 2500, 2003 and I'm trying to get the top 5 words from each document. How would you transform something that looks like this

(0, 1)  1

(0, 2)  1

(0, 6)  1

(1, 5)  2

(1, 1)  1

(2, 4)  1

(2, 7)  1

(3, 1)  1

etc into a count of the words by document? Thanks!

Answer 1

You need to find 5 sorted indices in each row of a sparse matrix. Please see similar questions: Scipy.sparse.csr_matrix: How to get top ten values and indices?

Finding the top n values in a row of a scipy sparse matrix

You can convert your sparse matrix to dense and use something like np.argsort(arr, axis=1) but it would be very inefficient in case of large matrices.

So the more efficient way is to implement it yourself:

import numpy as np
from scipy import sparse
from sklearn.feature_extraction.text import CountVectorizer


def top_k_sparse(mat: sparse.csr_matrix, n: int = 5):
    top_indices = []
    top_values = []
    for left, right in zip(mat.indptr[:-1], mat.indptr[1:]):
        n_row_pick = min(n, right - left)
        # At first use argpartition as it will be faster on large arrays
        row_top = np.argpartition(mat.data[left:right], -n_row_pick)[-n_row_pick:]
        # Reorder top indices
        row_top = row_top[np.argsort(-mat.data[left:right][row_top])]

        top_indices.append(mat.indices[left + row_top])
        top_values.append(mat.data[left + row_top])

    return top_indices, top_values


documents = [
    "qwe asd zxc a5 a5 a5 a5 a5 b3 b3 b3 c4 c4 c4 c4 fgf rtrt e2 e2 d2 d2",
    "a5 a5 a5 a5 a5 b3 b3 b3 c4 c4 c4 c4 fgf rtrt e2 e2 qwe asd zxc d2 zxc",
]
vectorizer = CountVectorizer()
sparse_mat = vectorizer.fit_transform(documents)
top_indices, top_values = top_k_sparse(sparse_mat, 5)

inverse_vocabulary = {v: k for k, v in vectorizer.vocabulary_.items()}
for row_indices, row_counts in zip(top_indices, top_values):
    words = [inverse_vocabulary[idx] for idx in row_indices]
    print(words, row_counts)

I used slightly modified implementation by Louis Yang from here .

Also if you need just top words you may consider using collections.Counter :

from collections import Counter

top_words = [Counter(doc.split(" ")).most_common(5) for doc in documents]
print(top_words)