So I've the following table with each row is a document and each column is words and no of occurrence of the words.
|doc|apple|banana|cat|
|---|---|---|---|
|1|2|0|0|
|2|0|0|2|
|3|0|2|0|
Is there any method to convert these count vectorized table to tf-idf vectorizer?
Edit: My solution for it. Let me know if this is correct.
def get_tfidf(df_tfidf):
total_docs = df_tfidf.shape[0]
#Term Frequency
#(Number of times term w appears in a document) / (Total number of
#terms in the document)
total_words_doc = df_tfidf.astype(bool).sum(axis=1)
tf = df_tfidf.values/total_words_doc[:,None]
#Inverse document frequency
#log_e(Total number of documents / Number of documents with term w in
#it)
words_in_doc = df_tfidf.astype(bool).sum(axis=0)
idf = np.log(total_docs/words_in_doc)
tf_idf = tf*idf.values[None,:]
return tf_idf
Suppose you have a Count Vectorizer as a pandas.DataFrame
like this:
import pandas as pd
data = [[1,2,0,0],[2,0,0,2],[3,0,2,0]]
df = pd.DataFrame(data,columns=['doc','apple','banana','cat'])
df
Output :
doc apple banana cat
0 1 2 0 0
1 2 0 0 2
2 3 0 2 0
Then you can use sklearn.feature_extraction.text.TfidfVectorizer
to get the tf-idf vector like this:
from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
x = v.fit_transform(df)
df1 = pd.DataFrame(x.toarray(), columns=v.get_feature_names())
print(df1)
Output :
apple banana cat doc
0 0.0 0.0 0.0 1.0
1 1.0 0.0 0.0 0.0
2 0.0 1.0 0.0 0.0
3 0.0 0.0 1.0 0.0
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.