如何使用 Scikit Learn 在语料库中获取单词/术语频率？

Question

I have a corpus of documents and I'd like to extract the word frequencies in each document. I could use CountVectorizer() to get term counts per document, and I could use TfidfVectorizer() to get term frequency-inverse document frequency, but neither seems to give me term frequencies alone. How do I get term frequencies?

This related question seems to ask my question, but the question and answers there concern term counts, not term frequencies. Maybe I'm the one misunderstanding these terms, but my understanding is that term counts are the integer number of times each term appears in the document whereas term frequencies are the term counts divided by the document length.

Answer 1

There is the TfidfTransformer for this purpose. From the docs:

Transform a count matrix to a normalized tf or tf-idf representation

Since it only transforms a count matrix, you would need to use it in conjunction with an already vectorized matrix or use CountVectorizer before:

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer


X_count = CountVectorizer().fit_transform(X_train)  # use first if X_train is not vectorized
X_tf = TfidfTransformer(use_idf=False).fit_transform(X_count)

Note that by setting use_idf=False you will get the term-frequency only.