从计数数据帧开始使用scikit学习tfidf矢量化器

Question

I have a pandas data frame with counts of words for a series of documents. Can I apply sklearn.feature_extraction.text.TfidfVectorizer to it to return a term-document matrix?

import pandas as pd

a = [1,2,3,4]
b = [1,3,4,6]
c = [3,4,6,1]

df = pd.DataFrame([a,b,c])

How can I get tfidf version of counts in df?

Answer 1

like this:

from sklearn.feature_extraction.text import TfidfTransformer
tfidf =TfidfTransformer(norm=u'l2', use_idf=True, smooth_idf=True, sublinear_tf=False)
data =tfidf.fit_transform(df.values)

This returns a sparse matrix of the tfidf values. You can turn them into a dense and put them back into a data frame like this:

pd.DataFrame(data.todense())