如何获得 CountVectorizer ngram 频率

Question

I have a dataset of ~50k short texts, average 9 tokens each. They contain a large number of uncommon tokens ('nw', '29203822', 'x989', etc...) as well as regular words and I believe these are degrading my classification efforts. I want to generate a stop word list of the most frequent n-grams that offer no value and remove them. I figure the best way is after my Count Vectorizer but before my TF-IDF to get those counts.

count_vect = CountVectorizer(ngram_range=(1,4))
X_train_counts = count_vect.fit_transform(X_train)
X_train_counts.shape

(19859, 122567)

count_vect.vocabulary_

{'look': 66431,
'1466': 1827,
'cl sign': 23055,
'in': 56587,
...}

I don't see any function for outputting the frequency of these ngrams within the dataset. Is there any? Thanks!

Answer 1

There is no build function for that (as far as I know), but you can achieve it with following function:

def create_n_gram_frequency(n_gram_from, n_gram_to, corpus):
    vec = CountVectorizer(ngram_range=(n_gram_from, n_gram_to)).fit(corpus)
    bag_of_words = vec.transform(corpus)
    sum_words = bag_of_words.sum(axis = 0)
    words_freq = [(word, sum_words[0, i]) for word, i in vec.vocabulary_.items()]
    words_freq = sorted(words_freq, key = lambda x: x[1], reverse = True)
    return words_freq