在python中检查sklearn的tf-idf分数

Question

I am following the example here to calculate the TF-IDF values using sklearn.

My code is as follows.

from sklearn.feature_extraction.text import TfidfVectorizer
myvocabulary = ['life', 'learning']
corpus = {1: "The game of life is a game of everlasting learning", 2: "The unexamined life is not worth living", 3: "Never stop learning"}
tfidf = TfidfVectorizer(vocabulary = myvocabulary, ngram_range = (1,3))
tfs = tfidf.fit_transform(corpus.values())

I want to calculate the tf-idf values for the two words life and learning for the 3 documents in corpus .

According to the article I am referring (see Table below) I should get the following values for my example.

However, the values I get from my code is completely different. Please help me find what is wrong in my code and how to fix it.

Answer 1

The main point is that you should not restrict the vocabulary to just two words ('life', 'learning') before constructing the term frequency matrix. If you do that, all other words will be ignored and it will affect the term frequency counting.

There are also several other steps that need to be taken into account if one wants to get exactly the same numbers as in the example by using sklearn:

1. The features in the example are unigrams (single words) so I have set ngram_range=(1,1) .

2. The example uses different normalization than sklearn for the term frequency part (the term counts are normalized by document lengths in the example, whereas sklearn uses raw term counts by default). Because of this, I have counted and normalized the term frequencies separately before calculating the idf part.

3. The normalization in the example for the idf part is also not the default for sklearn. This can be adjusted to match the example by setting smooth_idf to false.

4. Sklearn's vectorizers discard by default words with just one character, but such words are kept in the example. In the code below, I have modified token_pattern to allow also 1-character words.

The final tfidf matrix is obtained by multiplying the normalized counts by the idf vector.

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.preprocessing import normalize
import pandas as pd

corpus = {1: "The game of life is a game of everlasting learning", 2: "The unexamined life is not worth living", 3: "Never stop learning"}

cvect = CountVectorizer(ngram_range=(1,1), token_pattern='(?u)\\b\\w+\\b')
counts = cvect.fit_transform(corpus.values())
normalized_counts = normalize(counts, norm='l1', axis=1)

tfidf = TfidfVectorizer(ngram_range=(1,1), token_pattern='(?u)\\b\\w+\\b', smooth_idf=False)
tfs = tfidf.fit_transform(corpus.values())
new_tfs = normalized_counts.multiply(tfidf.idf_)

feature_names = tfidf.get_feature_names()
corpus_index = [n for n in corpus]
df = pd.DataFrame(new_tfs.T.todense(), index=feature_names, columns=corpus_index)

print(df.loc[['life', 'learning']])

However, in practice such modifications are rarely needed. One usually obtains good results just by using TfidfVectorizer directly.