如何獲取語料庫中某個詞的平均 TF-IDF 值？

Question

我正在嘗試獲取整個語料庫中某個單詞的平均 TF-IDF 值。 假設我們在我們的語料庫（幾百個文檔）中出現了 4 次“stack”這個詞。 它在找到的 4 個文檔中具有這些值0.34, 0.45, 0.68, 0.78 。 因此，整個語料庫的平均 TF-IDF 值為0.5625 。 我如何為文檔中的所有單詞找到這個？

我正在使用 TF-IDF 的 scikit-learn 實現。 這是我用來獲取每個文檔的 TF-IDF 值的代碼：

for i in docs_test:
    feature_names=cv.get_feature_names()

    doc=docs_test[itr]
    itr += 1
    tf_idf_vector=tfidf_transformer.transform(cv.transform([doc]))

    sorted_items=sort_coo(tf_idf_vector.tocoo())

    #Extracting the top 81 keywords along with their TF-IDF scores
    keywords=extract_topn_from_vector(feature_names,sorted_items,81)

對於每次迭代，這將輸出一個包含 81 個單詞的字典及其該文檔的 TF-IDF 分數： {'kerry': 0.396, 'paris': 0.278, 'france': 0.252 ......}

因為我只輸出前 81 個單詞，所以我知道該文檔中的所有單詞都不會被覆蓋。 所以，我想要文檔中前 81 個單詞中每個單詞的平均 TF-IDF 值（單詞將被重復）。

編輯：我嘗試了@mijjiga 的解決方案。 結果如下：

{'the': 0.51203095036175, 'to': 0.36268858983957286, 'of': 0.3200193439760937, 'in': 0.256015475180875, 'he': 0.2133462293173958}
{'the': 0.5076730825668095, 'to': 0.3299875036684262, 'in': 0.3299875036684262, 'and': 0.30460384954008574, 'trump': 0.17768557889838335}
{'the': 0.5257856140532874, 'children': 0.292103118918493, 'to': 0.2336824951347944, 'winton': 0.2336824951347944, 'of': 0.2336824951347944}
{'the': 0.6082672845890075, 'to': 0.3146210092701763, 'trump': 0.2936462753188312, 'that': 0.23911196704533397, 'of': 0.21394228630371986}
{'the': 0.6285692218670833, 'to': 0.3610929572427925, 'of': 0.2139810116994326, 'that': 0.20060719846821806, 'iran': 0.18723338523700353}
{'the': 0.5730922466510651, 'clinton': 0.29578954665861423, 'of': 0.24032900666012408, 'in': 0.2218421599939607, 'that': 0.2218421599939607}
{'the': 0.7509270472649924, 'to': 0.34926839407674065, 'trump': 0.17463419703837033, 'of': 0.17463419703837033, 'delegates': 0.1571707773345333}
{'on': 0.4, 'administration': 0.2, 'through': 0.2, 'the': 0.2, 'tax': 0.2}
{'the': 0.5885277950982455, 'in': 0.3184973949943446, 'of': 0.3046496821685035, 'to': 0.29080196934266245, 'women': 0.2769542565168214}

正如我們所見，“the”這個詞有多個值。 如果我的問題沒有表明這一點，我深表歉意，但我希望每個單詞都有一個值。 該值是該文檔語料庫中該單詞的平均 TF-IDF 分數。 有關如何使其工作的任何幫助？ 謝謝！

這是使用的代碼：

from sklearn.feature_extraction.text import TfidfVectorizer 
import numpy as np
itr = 0
for i in range(1,10):
    docs=docs_test[itr]
    docs=[docs]
    itr+=1
    tfidf_vectorizer=TfidfVectorizer(use_idf=True)
    tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(docs)

    tfidf = tfidf_vectorizer_vectors.todense()
    # TFIDF of words not in the doc will be 0, so replace them with nan
    tfidf[tfidf == 0] = np.nan
    # Use nanmean of numpy which will ignore nan while calculating the mean
    means = np.nanmean(tfidf, axis=0)
    # convert it into a dictionary for later lookup
    means = dict(zip(tfidf_vectorizer.get_feature_names(), means.tolist()[0]))

    tfidf = tfidf_vectorizer_vectors.todense()
    # Argsort the full TFIDF dense vector
    ordered = np.argsort(tfidf*-1)
    words = tfidf_vectorizer.get_feature_names()

    top_k = 5
    for i, doc in enumerate(docs):
        result = { }
        # Pick top_k from each argsorted matrix for each doc
        for t in range(top_k):
            # Pick the top k word, find its average tfidf from the
            # precomputed dictionary using nanmean and save it to later use
            result[words[ordered[i,t]]] = means[words[ordered[i,t]]]
        print (result )

Answer 1

文檔是內聯的

from sklearn.feature_extraction.text import TfidfVectorizer 
import numpy as np

docs=["the house had a tiny little mouse",
      "the cat saw the mouse",
      "the mouse ran away from the house",
      "the cat finally ate the mouse",
      "the end of the mouse story"
     ]

tfidf_vectorizer=TfidfVectorizer(use_idf=True)
tfidf_vectorizer_vectors=tfidf_vectorizer.fit_transform(docs)

tfidf = tfidf_vectorizer_vectors.todense()
# TFIDF of words not in the doc will be 0, so replace them with nan
tfidf[tfidf == 0] = np.nan
# Use nanmean of numpy which will ignore nan while calculating the mean
means = np.nanmean(tfidf, axis=0)
# convert it into a dictionary for later lookup
means = dict(zip(tfidf_vectorizer.get_feature_names(), means.tolist()[0]))

tfidf = tfidf_vectorizer_vectors.todense()
# Argsort the full TFIDF dense vector
ordered = np.argsort(tfidf*-1)
words = tfidf_vectorizer.get_feature_names()

top_k = 5
for i, doc in enumerate(docs):
    result = { }
    # Pick top_k from each argsorted matrix for each doc
    for t in range(top_k):
        # Pick the top k word, find its average tfidf from the
        # precomputed dictionary using nanmean and save it to later use
        result[words[ordered[i,t]]] = means[words[ordered[i,t]]]
    print (result )

輸出

{'had': 0.4935620852501244, 'little': 0.4935620852501244, 'tiny': 0.4935620852501244, 'house': 0.38349121689490395, 'mouse': 0.24353457958557367}
{'saw': 0.5990921556092994, 'the': 0.4400321635416817, 'cat': 0.44898681252620987, 'mouse': 0.24353457958557367, 'ate': 0.5139230069660121}
{'away': 0.4570928721125019, 'from': 0.4570928721125019, 'ran': 0.4570928721125019, 'the': 0.4400321635416817, 'house': 0.38349121689490395}
{'ate': 0.5139230069660121, 'finally': 0.5139230069660121, 'the': 0.4400321635416817, 'cat': 0.44898681252620987, 'mouse': 0.24353457958557367}
{'end': 0.4917531872315962, 'of': 0.4917531872315962, 'story': 0.4917531872315962, 'the': 0.4400321635416817, 'mouse': 0.24353457958557367}

讓我們解密result[words[ordered[i,t]]] = means[words[ordered[i,t]]]

如果詞匯量是v並且文檔數是n那么

• ordered是大小為nxv的矩陣。 該矩陣的值是對應於詞匯的索引，並且該矩陣根據每個文檔的 TF-IDF 分數進行排序。
• words是words表中words的列表大小v 。 將此視為單詞映射器的 id
• means是大小為一個字典v與每個值是所述字的平均TF-IDF。