如何通过scikit和pandas获得带有频率的Ngram排名列表？

Question

I am trying to this simple task with scikit but I am having trouble working with the sparse matrix. For this, I don't care about document frequency.

This is what I have so far:

vectorizer = CountVectorizer(ngram_range=(1,3))
n_grams = vectorizer.fit_transform(df.column_with_text)

At this point I know I am supposted to do something involving n_grams and inverse_transform , but I'm not sure what. I would like a list of [n_gram,frequency] ranked by frequency, like this:

"apple banana", 100
"this is fun", 100
"cool pandas", 99
...

Thanks.

Answer 1

You get the vocabulary out of your vectoriser with vocabulary_ ; the values are the columns of the vectorized output corresponding to the keys:

vectorizer.vocabulary_
{'apple': 0,
 'apple banana': 1,
 'apple banana this': 2,

The frequencies will be the sums of the columns of n_grams , to calculate these it's probably easiest to convert the sparse matrix to a numpy array first with toarray() , then one way to match them up would be with a list comprehension:

vocab = vectorizer.vocabulary_
count_values = n_grams.toarray().sum(axis=0)
counts = sorted([(count_values[i],k) for k,i in vocab.items()], reverse=True)

counts
[(4, 'pandas'),
 (4, 'cool pandas'),
 (4, 'cool'),
 (2, 'this is fun'),
 (2, 'this is'),